Semantic Gap in Predicting Mental Wellbeing through Passive Sensing

Document Title

Semantic Gap in Predicting Mental Wellbeing through Passive Sensing

Document Information

• Topic Area: Mental health prediction and passive sensing
• Keywords: Passive sensing, mental health, social media, activity patterns, semantic gap, physiological measurement, self-report, multimodal sensing

Research Background and Problem Statement

• Identified Problems or Challenges:
  Passive sensing technology for mental health inference often relies on self-reports as ground truth labels. However, self-reports primarily reflect psychological aspects and cannot fully align with physiological states, potentially leading to a "semantic gap" between the two dimensions. Additionally, passive sensing data struggles to capture the complete ecological factors influencing ground truth labels.

• Importance of the Study:
  Mental health status is closely tied to individual wellbeing and productivity. Understanding the semantic gap can advance the application of passive sensing technology, providing efficient and accurate predictive tools for the mental health domain.

• Motivation and Related Work:
  The semantic gap has been widely discussed in computational fields such as natural language processing and computer vision, but its impact on mental health prediction has not been systematically explored. This study introduces the semantic gap issue in passive sensing data and attempts to evaluate its prevalence and resolution paths using multimodal data.

Solution

• Proposed Method or Solution:
  Using multimodal data (three offline sensor streams and social media language features) to predict two major measurement dimensions of mental health ground truth labels: self-reports (psychological dimension) and high arousal states in physiological measurements (physiological dimension). Two core research questions are proposed:

  • RQ1: Are social signals better predictors of self-reports (psychological dimension) compared to behavioral signals?
  • RQ2: Are behavioral signals better predictors of high arousal states (physiological dimension) compared to social signals?

• Innovations:

  1. First application of the "semantic gap" theory to mental health prediction.
  2. Utilization of cross-sensor data (offline sensors and social media language) to represent different abstraction dimensions of mental health constructs.
  3. Quantitative comparison of semantic gaps using multimodal data.

• Implementation Steps and Key Technologies:

  1. Application of multimodal sensor data (smartphones, wearable devices, Bluetooth devices) and social media language features.
  2. Use of regression methods (Random Forest, Gradient Boosting, XGBoost) to estimate mental health constructs.
  3. Analysis of the impact of different window lengths on model prediction performance, comparing models using correlation and Symmetric Mean Absolute Percentage Error (SMAPE).
  4. Post-hoc analysis to validate the enhancement of self-report prediction by social signals and test the predictive effectiveness of behavioral signals for physiological measurements.

Research Outcomes

• Specific Findings:

  1. Social media language is more effective than offline behavioral signals in predicting anxiety and stress in self-reports (psychological dimension).
  2. Behavioral signals are more accurate than social language signals in predicting high arousal states in physiological measurements (physiological dimension).
  3. Adding simulated social signals (Bluetooth interaction features) significantly improves behavioral signal-based prediction models.

• Advantages Compared to Existing Solutions:

  1. Provides sensor selection recommendations tailored to different mental health constructs (psychological and physiological).
  2. Advocates for resource optimization through "minimal sensing" by reducing sensor quantity.
  3. Expands understanding of the sensitivity and characteristics of ground truth labels in the passive sensing domain.

• Experimental or Evaluation Results:

  • For self-reports in the psychological dimension, the correlation of social media language models reached r = 0.56, outperforming behavioral signal models with r = 0.34.
  • For high arousal states in the physiological dimension, the correlation of behavioral signal models reached r = 0.63, outperforming social media models with r = 0.56.
  • Adding Bluetooth-derived interaction features improved the correlation of behavioral signal predictions for self-reports by approximately 64%.

• Limitations and Future Directions:

  1. Dataset limitation: The study is based solely on a specific dataset (Tesserae project), and results may not generalize to other datasets.
  2. Scope of constructs: Only anxiety and stress were studied; future research could expand to other mental health constructs.
  3. Model types: Focused on dynamic states as target variables; future work could explore stable psychological constructs (e.g., social impairments).
  4. Quality control: Further exploration of contextual influences on self-reports is suggested to filter more reliable ground truth labels.