Predicting early user churn in a public digital weight loss intervention

Title of the Paper

Predicting Early User Attrition in Public Digital Weight Loss Interventions

Paper Information

• Research Domain: Digital Health, Machine Learning, Behavioral Prediction
• Keywords: Machine Learning, Attrition Prediction, User Retention, mHealth, Digital Health, Weight Management, Behavioral Intervention, Multidimensional Features

Research Background and Problem

• Problems and Challenges:

  1. Digital Health Interventions (DHIs) often experience high attrition rates, preventing users from achieving expected health outcomes.
  2. Attrition rates are particularly high in publicly available weight management applications, with only about 42.2% of users meeting post-trial usage requirements.
  3. Most attrition prediction studies focus on controlled environments, making it difficult to generalize findings to public health applications. Additionally, many studies predict attrition after users have already disengaged, rather than at an early stage.
  4. Few studies explore the effectiveness of re-engaging users after predicting attrition.

• Significance:
  High attrition rates not only hinder the effectiveness of health interventions but also lead to wasted resources and significantly reduced user satisfaction. Accurate early prediction of user attrition can optimize personalized interventions to improve retention rates and enhance health outcomes.

• Research Motivation and Related Work:
  The authors aim to address critical gaps in existing research, including leveraging machine learning (ML) algorithms to predict early attrition, analyzing the impact of multidimensional features, and evaluating the potential for re-engaging users. The authors highlight that prior studies, such as Kwon et al., explored attrition in publicly available weight loss applications but used overly long data windows and neglected early-stage attrition.

Solution

• Methods or Solutions:

  1. Predict attrition in a publicly available subscription-based weight management application (WayBetter) and explore attrition dynamics during the 7-day trial period.
  2. Experimental data includes 310,845 event logs from 1,283 users, employing various ML algorithms such as Logistic Regression, Decision Trees, Random Forest (RF), XGBoost, and Artificial Neural Networks (ANN).
  3. Compare models with different feature dimensions, including low-dimensional (LDM, primarily based on log frequency), medium-dimensional (MDM, integrating independent variables like installation date and user weight), and high-dimensional (HDM, encompassing 174 in-app event features).

• Innovations:

  1. First study to implement 7-day attrition prediction using different model dimensions and ML algorithms on synthesized real user engagement data.
  2. Investigate which features contribute most to attrition prediction and analyze the potential of intervention strategies based on actual user behavior.
  3. Utilize Bayesian Logistic Regression (BLR) and other models to evaluate their applicability for Just-In-Time Adaptive Interventions (JITAIs).

• Implementation Steps:

  1. Data cleaning and preprocessing: Remove direct event logs reflecting subscription cancellations and handle outliers in the dataset.
  2. Feature construction: Include metrics such as daily login frequency, session duration, and user weight.
  3. Model training and evaluation: Train models from low-dimensional to high-dimensional, using 10-fold cross-validation and hyperparameter tuning to optimize key performance metrics (F1 score, AUC, etc.).

Research Outcomes

• Specific Findings:

  1. Despite a high attrition rate of 65%, the RF low-dimensional model achieved the best performance on Day 7, with an F1 score of 0.865 and a weekly average of 0.839, outperforming some high-dimensional models.
  2. The low-dimensional model, using only daily login frequency, achieved high-accuracy attrition prediction, offering advantages in simplicity and interpretability compared to high-dimensional models.
  3. High-dimensional models demonstrated stronger discrimination between positive and negative classes under low false-positive conditions, with a significantly higher average AUC value (0.789).

• Comparative Advantages:

  1. Compared to existing studies (e.g., Kwon et al., Bricker et al.), this study's simplified models achieved comparable performance, suggesting that daily login frequency alone is sufficient to capture user engagement dynamics.
  2. The authors further quantified the likelihood of re-engaging users through additional intervention measures, even with accurate predictions.

• Experimental or Evaluation Results:

  1. Approximately 93% of attrition users during the trial period were correctly identified by the models.
  2. The RF model accurately predicted an average of 80.6% of re-engaged users from Day 1 to Day 6.
  3. The effectiveness of attrition interventions decreased as users continued to disengage—re-engagement rates dropped from 48.8% on Day 2 to 18.6% on Day 7.

• Limitations and Future Directions:

  1. The definition of attrition for subscription-based models may not apply to freemium (mFreemium) applications, limiting the generalizability of the results.
  2. Rule-based in-app interventions (e.g., push notifications) influenced model performance but were not accounted for in adjustments.
  3. Future work should incorporate reinforcement learning to enhance dynamic adaptability of the models and validate intervention strategies through randomized controlled trials.