Designing an Affective Mobile Probe to Measure Smile Dynamics in Depression

Paper Title

Designing an Affective Mobile Probe to Measure Smile Dynamics in Depression

Publication Info

• Topic area: Behavioral markers of depression using mobile health technology
• Keywords: Depression, Duchenne smiles, mobile health, facial expressions, digital biomarkers, emotional reactivity, longitudinal study, mixed-effects modeling, anhedonia, affective computing

Background and Problem

• Problem / challenge: Traditional depression assessments rely heavily on self-reported surveys, which may not adequately capture behavioral symptoms like blunted emotional reactivity and anhedonia. Existing studies on facial markers of depression often lack ecological validity and focus on static or controlled environments.
• Significance: Identifying objective, behavioral markers of depression can improve remote monitoring and intervention, particularly in mobile health (mHealth) settings, where accessibility and scalability are key.
• Motivation and related work: Prior work, such as MoodCapture and FacePsy, has demonstrated the potential of facial expressions for depression prediction but has focused on static images or non-contextualized data. This study builds on these efforts by analyzing dynamic facial expressions in a longitudinal, ecologically valid setting.

Solution

• Proposed approach: Affective Mobile Probe (AMP), a smartphone-based system that captures and analyzes smile dynamics in response to humorous video stimuli to assess depression-related emotional reactivity.
• Novelty:
  1. Introduction of a novel AMP paradigm to measure dynamic facial expressions in real-world settings.
  2. Differentiation of within-individual and between-individual effects using mixed-effects modeling.
  3. Analysis of smile dynamics (e.g., Duchenne smiles, smirks) as candidate markers of depression severity.
  4. Integration of user feedback to refine the design and usability of AMP systems.
• Procedure and key techniques:
  1. Participants watch humorous baby videos on their smartphones while their facial expressions are recorded.
  2. Facial action units (e.g., AU6, AU12) are extracted using Affdex 2.0 software.
  3. Smile dynamics (e.g., intensity, velocity, time to peak) are analyzed using functional data analysis and mixed-effects models.
  4. Depression severity is assessed using PHQ-8 scores, and associations with smile dynamics are modeled longitudinally.

Results

• Concrete findings:
  • Higher Duchenne smile intensity and faster onset in response to liked videos are significantly associated with lower PHQ-8 depression scores (e.g., within-individual effect size for Duchenne top percentile: -0.051).
  • Smirk dynamics show weaker and less consistent associations with depression.
  • The strongest associations are observed when participants find the videos funny, aligning with emotion context insensitivity (ECI) theory.
  • The AMP paradigm explains 8% of the variance in PHQ-8 scores, with most variability attributed to stable between-person differences.
• Advantage over baselines: The study demonstrates the feasibility of detecting depression-linked emotional reactivity in real-world, mobile settings, complementing prior work that relied on static or less ecologically valid contexts.
• Experiments / evaluation:
  • Dataset: 684 participants, 2,702 sessions over 16 weeks from the BeWell trial.
  • Metrics: PHQ-8 depression scores, smile intensity, velocity, and time to peak.
  • Methods: Mixed-effects modeling, bootstrap analysis, and BERTopic for user feedback analysis.
• Limitations and future work:
  • Limited generalizability to underrepresented demographic groups due to small sample sizes.
  • Modest effect sizes and predictive power.
  • Need for more diverse stimuli and increased sampling density.
  • Future work should integrate AMP with other sensing modalities and explore personalized stimuli.

Summary

This study introduces the Affective Mobile Probe (AMP), a smartphone-based system that measures smile dynamics as behavioral markers of depression. Longitudinal analysis of 684 participants shows that higher Duchenne smile intensity in response to liked humorous videos is associated with lower depression severity, supporting emotion context insensitivity (ECI) theory. While the results highlight the potential of AMP for remote depression monitoring, limitations in demographic representation and predictive power suggest the need for further refinement, including personalized stimuli and integration with other sensing modalities. The findings underscore the importance of ecological validity and user-centered design in mHealth interventions.