Exploring Context-Aware Mental Health Self-Tracking Using Multimodal Smart Speakers in Home Environments

Title of the Paper

Exploring Context-Aware Mental Health Self-Tracking Using Multimodal Smart Speakers in Home Environments

Paper Information

• Topic Area: Mental health self-tracking using multimodal smart speakers
• Keywords: self-tracking, mental health, multimodal smart speakers, Experience Sampling Method (ESM), home environment, IoT sensing

Research Background and Issues

• Issues and Challenges:
  • Traditional healthcare lacks the ability to monitor dynamic changes in mental health frequently.
  • Mobile devices (e.g., phones, smartwatches) used for ESM face challenges of insufficient carryability at home.
  • Existing voice-based smart speaker methods lack exploration of multimodal user experience.
• Significance: Mental health issues have become a global concern. Long-term tracking can more accurately reflect dynamic psychological states and support interventions.
• Research Motivation and Related Work:
  • Smart speakers are widely introduced in home environments, but their multimodal interaction design remains underexplored.
  • The Experience Sampling Method (ESM) can capture real-time fluctuations in mental health, but research on trigger timing and user interaction modes is limited.

Solution

• Method or Solution:
  • Design and develop an IoT-sensing-based multimodal smart speaker system for mental health self-tracking, integrating voice and touch interactions.
  • The system triggers context-aware ESM questionnaires using sound and light sensors, CO2 sensors, and cameras, allowing users to respond via voice or touch.
• Innovations:
  • Investigate user experience in multimodal interactions (voice and touch) for mental health self-tracking.
  • Utilize environmental sensors to detect user behavior changes and optimize the timing of ESM task triggers.
  • Collect long-term user data in home settings to study preferences and reactions in real-world scenarios.
• Implementation Steps:
  1. Design a sensor system to perceive user context (e.g., noise level changes, light variations, CO2 concentration changes).
  2. Develop an ESM system with voice guidance and touch options.
  3. Use multimodal smart speakers and Google Nest Hub to collect user responses to mental health questionnaires.
  4. Conduct a four-week field test, recruiting participants with mild depressive symptoms to distribute the system.
  5. Analyze user data using interviews, questionnaires (USE), and multilayer logistic regression.

Research Outcomes

• Specific Outcomes:
  1. User Engagement and Compliance:
     • Context-aware sensor-triggered ESM improved user response rates (overall response rate 57%).
     • Response rates reached 76.2% under external context triggers such as light and human presence.
  2. Interaction Mode Preferences:
     • 93.8% of users preferred completing questionnaires via touch due to shorter interaction time and higher technical accuracy.
     • During multitasking activities (e.g., housework, personal hygiene), users preferred voice interactions.
  3. User Experience Feedback:
     • The system encouraged self-reflection on mental health, especially by actively reminding users to focus on emotional fluctuations.
     • The multimodal design's "human-like" qualities enhanced user immersion.
     • Monotonous voice content reduced interaction novelty.
  4. Privacy and Technical Issues:
     • Despite cameras being used solely for detecting the number of people and not saving images, users still expressed concerns about privacy risks.
     • Some users were dissatisfied with Google Assistant's voice recognition errors.
• Advantages Compared to Existing Solutions:
  • The system integrates IoT environmental sensing technology, improving the timeliness and personalization of mental health questionnaire triggers.
  • The system design considers users' activity distribution and technology usage habits in home environments.
• Experimental or Evaluation Results:
  • Users generally found the system easy to use (USE questionnaire score: learnability M=6.44), with high ratings for operational convenience and satisfaction (M=4.64).
  • Rich activity options and interactive system feedback significantly enhanced user experience.
• Limitations and Future Directions:
  • Limitations:
    • Limited to single-person household scenarios: does not address compatibility in multi-person households.
    • Voice recognition system lacks sufficient accuracy.
    • Fixed questionnaire content led to user fatigue, potentially affecting data quality.
  • Future Directions:
    • Explore context-aware interactions in multi-user household environments, integrating wearable devices and mobile interactions to optimize precision.
    • Develop more intelligent voice and image privacy protection mechanisms.
    • Further study visual data presentation to stimulate user feedback behavior.