Affective State Prediction from Smartphone Touch and Sensor Data in the Wild

Title of the Paper

Affective State Prediction from Smartphone Touch and Sensor Data in the Wild

Paper Information

• Research Area: Human-Computer Interaction and Affective Computing
• Keywords: Affective Prediction, Smartphone Sensors, Touchscreen Data, Deep Learning, Data Privacy

Research Background and Problem

• Problems or Challenges Identified by the Authors: Smartphones, as everyday devices, can collect a large amount of user interaction data. However, effectively utilizing this data for predicting users' affective states while ensuring privacy protection remains a significant research challenge.

• Why This Problem Is Important: Predicting affective states can enhance user experience and has significant applications in scenarios such as health management, education, and personalized recommendations. For example, it can help users alleviate psychological issues or optimize recommendation systems based on personal emotional dynamics.

• Research Motivation and Related Work: Existing studies on affective state prediction are often conducted in laboratory settings, where emotional data is collected using stimuli like videos or images. These settings lack ecological validity compared to real-world usage scenarios. The authors highlight the need to develop affective state prediction models based on real-world smartphone interaction data, focusing on affective dimensions such as valence, arousal, and dominance.

Solution

• Methods or Solutions Proposed by the Authors: The authors developed an affective state classification model that generates two-dimensional heatmaps from smartphone touch interactions and inertial sensor data (gyroscope and accelerometer). These heatmaps are then used for affective state classification via a Convolutional Neural Network (CNN).

• Innovative Aspects of the Solution:

  1. Real-World Data Collection: Conducted a 10-week study with 82 participants, collecting a large dataset of "in-the-wild" smartphone interactions (30,083 self-reports).
  2. Privacy-Friendly Design: Achieved classification accuracy close to that of touchscreen data by using only sensor data, reducing risks to user privacy.
  3. Efficient Model Execution: Proposed a compact CNN model based on MobileNetV2, enabling efficient operation on mobile devices with accurate predictions.
  4. Personalization Enhancement: Incorporated user-specific adjustments to improve prediction performance.

• Implementation Steps and Key Techniques:

  1. Data Collection: Developed an Android app to record touchscreen and sensor data, along with user self-reports.
  2. Feature Extraction: Generated two-dimensional heatmaps (including typing speed and sensor data frequency distribution).
  3. Model Design: Used MobileNetV2 to extract heatmap features and a fully connected classification layer for three-class classification (low, medium, high affective states).
  4. Model Personalization: Fine-tuned the model using individual user reports to further enhance prediction performance.

Research Outcomes

• Specific Achievements:

  1. Predicted valence, arousal, and dominance using touchscreen heatmaps with AUCs of 0.82, 0.81, and 0.82, respectively. Using only sensor data achieved AUCs of 0.79, 0.83, and 0.81, respectively. Combining both further improved AUCs to 0.83, 0.85, and 0.84.
  2. Achieved a maximum AUC of 0.86 for predicting basic emotions (e.g., happiness, sadness, anger) and 0.83 for stress prediction.

• Advantages Compared to Existing Solutions:

  1. High Accuracy: Outperformed traditional laboratory studies or affective inference methods in real-world scenarios.
  2. Simplicity: Represented data as heatmaps instead of processing raw sensor data directly, making the approach more intuitive and resource-efficient.
  3. Privacy Protection: Achieved good prediction performance without long-term text content recording, relying solely on sensor data.

• Experimental or Evaluation Results:

  • The model's robustness and prediction accuracy were validated through stratified cross-validation on the dataset.
  • Prediction performance improved by up to +0.07 AUC after user-specific adjustments.

• Limitations and Future Directions:

  1. Limitations:

     • Model performance is limited to specific participant groups (primarily young students), reducing generalizability.
     • Did not address affective state prediction in scenarios without typing (e.g., watching videos).
     • Runtime evaluation was conducted on high-performance computing environments, requiring further optimization for smartphone hardware.

  2. Future Directions:

     • Explore seamless integration of time-series data, such as through Long Short-Term Memory (LSTM) networks.
     • Extend heatmaps to three-dimensional features to capture more complex motion relationships.
     • Integrate contextual information, such as ambient lighting and app types, to further enhance model performance.
     • Validate model performance across a broader population and expand research into related areas (e.g., personality trait prediction).