Towards Estimating Missing Emotion Self-reports Leveraging User Similarity: A Multi-task Learning Approach

Title of the Paper

Towards Estimating Missing Emotion Self-reports Leveraging User Similarity: A Multi-task Learning Approach

Paper Information

• Research Domain: Affective Computing, User Behavior Modeling, Multi-task Learning
• Keywords: Experience Sampling Method (ESM), Emotion self-report, Multi-task learning, Missing data, User similarity, Data collection, Neural networks, Emotion inference

Research Background and Problem

• Identified Problem or Challenge: In emotion self-report studies based on the Experience Sampling Method (ESM), participants often drop out midway due to the prolonged and tedious nature of the study. This unplanned attrition results in reduced data volume and quality, which negatively impacts the reliability and applicability of research findings.
• Significance of the Problem: Emotion self-reports are a core data source for emotion inference models. Missing data increases the cost of repeating studies and reduces the performance of models in real-world tasks.
• Research Motivation and Related Work:
  • Efforts to prevent attrition include offering rewards or incentives, but these measures fail to completely eliminate participant dropout.
  • Existing approaches to impute missing data often rely on additional sensor data, which may raise privacy concerns or incur resource costs.
  • Current research lacks a general method to estimate missing emotion self-reports solely based on user similarity and shared self-report data.

Proposed Solution

• Proposed Solution: A multi-task learning (MTL) framework called MUSE is proposed, which leverages the similarity of emotion self-reports among different users to estimate the missing self-reports of dropout participants.
• Innovative Contributions:
  • Does not rely on sensor data, using only user self-report similarity.
  • Quantifies user emotion reporting behavior characteristics (e.g., emotion state transition probabilities, emotion duration, emotion repetition length).
  • Applies multi-task learning to improve model accuracy by sharing data across users.
• Implementation Steps and Key Techniques:
  1. Define three characteristics of emotion reporting behavior: emotion state transition probabilities, emotion duration, and emotion repetition length.
  2. Use these characteristics to describe each user's emotion reporting patterns and model them using a neural network-based multi-task learning framework.
  3. Reduce the dependency on individual user data by sharing task-relevant data across users.
  4. Train and validate the model using data from real participants to predict missing emotion self-reports.

Research Outcomes

• Specific Results:
  • MUSE was evaluated in two longitudinal studies, using data from 24 students (6 weeks) and 30 participants from diverse backgrounds (8 weeks).
  • The model achieved an average AUCROC of 84% on homogeneous datasets and 82% on heterogeneous datasets.
  • Monte Carlo sampling demonstrated that the estimated self-reports effectively supported downstream tasks such as emotion inference.
• Advantages over Existing Solutions:
  • Compared to sensor-dependent strategies, MUSE avoids privacy and resource cost issues by not relying on sensor data.
  • Multi-task learning significantly improves model performance by sharing data among similar users.
• Experimental or Evaluation Results:
  • The model performed well (AUCROC close to 80%) even when training data coverage was as low as 50%.
  • In real-world tasks (e.g., emotion detection based on smartphone keyboard interactions), substituting estimated data for original data did not significantly degrade performance.
  • The strong performance on heterogeneous datasets indicates that MUSE is highly adaptable and generalizable.
• Limitations and Future Directions:
  • The model's performance may decline in scenarios with shorter study durations or lower user response rates.
  • The current framework primarily addresses complete dropout scenarios and needs improvement for intermittent missing data (e.g., partial non-responses).
  • Scalability and generalizability need further validation in larger-scale and longer-term experiments.