Are You Killing Time? Predicting Smartphone Users’ Time-killing Moments via Fusion of Smartphone Sensor Data and Screenshots

Title of the Paper

Are You Killing Time? Predicting Smartphone Users’ Time-killing Moments via Fusion of Smartphone Sensor Data and Screenshots

Paper Information

• Domain: Human-Computer Interaction and Mobile Computing Technology
• Keywords: Time-killing behavior, Screenshots, Deep learning, Smartphone sensor data, Mobile usage

Research Background and Problem

• Problem or Challenge: While smartphone usage is not always driven by specific purposes, many people use their phones to kill time, such as during waiting or moments of boredom. However, research on detecting such behavior is currently limited. Existing studies primarily focus on detecting disruptive or attention-surplus moments, with little exploration into time-killing behavior.
• Importance of the Research: Detecting time-killing behavior not only facilitates more suitable content recommendations for users but also helps researchers analyze when and how frequently this behavioral pattern occurs throughout the day. This can improve smartphone usage habits and enhance productivity.
• Motivation and Related Work: Detecting time-killing behavior enables developers to focus product design and interventions on moments of low interaction, thereby enhancing application user experience and usage efficiency.

Solution

• Method or Solution: A deep learning model is proposed that integrates smartphone sensor data and screenshot data to detect users' time-killing behavior.
• Innovative Points of the Solution:
  1. Combining features from smartphone screenshots and sensor data to improve prediction accuracy.
  2. Grouping users based on their smartphone usage behavior and constructing separate prediction models for each group.
• Implementation Steps and Key Techniques:
  • Data Collection: Data was collected using a self-developed Android application, "Killing Time Labeling (KTL)," which gathered sensor data, screenshot data, and user annotations of time-killing behavior. Data from 36 participants was collected, totaling over 960,000 annotations.
  • Feature Extraction: Smartphone state and user interaction features were extracted from sensor data, while graphical features were extracted from screenshot images.
  • Model Design: Deep learning techniques were employed, including DeepFM for processing sensor data, ResNet for image feature extraction, and an attention mechanism for data integration, forming the final classification prediction.
  • User Grouping: A two-stage clustering algorithm based on k-means was used to group users according to their smartphone usage habits, with dedicated models constructed for each group.
  • Model Training: A phased training strategy was adopted, with pre-training of screenshot and sensor data encoders followed by joint training of the integrated network.

Research Results

• Specific Results:
  • The integrated model achieved an average precision of 0.83 and an AUROC of 0.72 in predicting time-killing behavior. Compared to models using only sensor or screenshot data, the integrated model performed better.
  • Models based on user behavior grouping further improved performance, achieving an average precision of 0.87 and an AUROC of 0.76, surpassing the unified general model.
• Advantages:
  • Compared to existing research, the integrated model demonstrated higher accuracy and specificity in predicting time-killing behavior, particularly in reducing the likelihood of non-time-killing behavior being misclassified as time-killing.
  • User grouping models improved robustness in cases of diverse data distributions.
• Limitations and Future Directions:
  • Limitations include the small sample size (only 36 participants) and single-source data (Taiwan region), which may restrict the model's generalizability.
  • User annotation behavior may be influenced by recall bias, affecting the reliability of data labeling.
  • Incomplete privacy protection measures may impact user behavior, as providing raw screenshots could pose privacy concerns for participants.
  • Future research directions include expanding the sample size to enhance model generalizability, optimizing privacy protection methods (e.g., blurring sensor and screenshot data), and exploring additional feature associations for time-killing behavior.

Summary and Contributions

• Summary: By employing a unique model that integrates screenshot and sensor data, researchers successfully detected smartphone users' time-killing behavior, paving the way for advancements in intelligent content recommendation and user behavior analysis. The modeling strategy based on user grouping demonstrated superior prediction capabilities.
• Academic Contributions:
  1. Proposed a deep learning model based on the integration of screenshot and sensor data for precise detection of time-killing behavior.
  2. Highlighted the importance of user behavior grouping in improving model performance.
  3. Provided an evaluation of the relative utility of sensor data and screenshot data across different user behavior patterns.