Exploring Smartphone Keyboard Interactions for Experience Sampling Method driven Probe Generation

Title of the Paper

Exploring Smartphone Keyboard Interactions for Experience Sampling Method driven Probe Generation

Paper Information

• Research Domain: Human-Computer Interaction, Emotion Sensing, and Intelligent User Interface Design
• Keywords: Smartphone, Keyboard Interaction, Typing, Experience Sampling Method (ESM), Opportune Moments, Notifications

Research Background and Problem

• Research Questions or Challenges:

  • The interaction between keyboard usage patterns (e.g., rhythm and latency) and emotion sensing services has not been thoroughly explored.
  • Collecting user emotion labels using the Experience Sampling Method (ESM) often leads to user fatigue and distraction, necessitating the design of timely and efficient survey strategies to improve response rates.
  • Existing studies primarily focus on user activity and context, overlooking potential patterns and features embedded in smartphone keyboard interaction data.

• Significance:

  • Research on smartphone keyboard interaction patterns can advance emotion sensing services, such as interface optimization, adaptive keyboards, and automatic emoji recommendations.
  • Optimizing ESM question timing in high-usage scenarios can reduce user interruptions and fatigue, thereby improving data quality.

• Motivation and Related Work:

  • Previous studies have explored identifying opportune notification moments based on phone activity, time, and context, but these approaches mainly rely on external factors and rarely incorporate user keyboard input behavior.
  • This study aims to leverage both time-domain and frequency-domain analyses to uncover the potential of keyboard interaction data for improving the accuracy and timeliness of ESM notifications.

Solution

• Proposed Method or Solution:

  • Utilize smartphone keyboard interaction patterns, including time-domain features (e.g., typing speed, session length, session duration) and frequency-domain features (e.g., number of peak amplitudes, peak magnitude), to identify suitable moments for ESM surveys.
  • Develop a general machine learning framework based on a Random Forest model that integrates time-domain and frequency-domain features to predict opportune survey moments.

• Innovations:

  • Systematically introduce frequency-domain analysis combined with rhythm features in keyboard interactions;
  • Design and implement a novel Android keyboard application that records only user touch events, not text content, to ensure privacy protection;
  • Apply model interpretability analysis (SHAP) to predict the timing of emotion self-reports.

• Implementation Steps and Key Techniques:

  1. Data Collection: Develop an Android keyboard app to track user keyboard interaction data while recording user-marked suitable and unsuitable moments in self-report questionnaires.
  2. Time-Domain Analysis: Calculate features such as session length, duration, typing speed, and error rate, and compare data differences between suitable and unsuitable moments.
  3. Frequency-Domain Analysis: Use Discrete Fourier Transform (DFT) to convert typing interval times into frequency-domain features and extract key peak amplitudes.
  4. Model Construction: Train a Random Forest model combining time-domain and frequency-domain features for classification.
  5. Model Interpretability Analysis: Use the SHAP method to quantify the contribution of each feature to the model's prediction accuracy.

Research Outcomes

• Specific Findings:

  • Time-domain features revealed that, compared to unsuitable moments, suitable moments were characterized by significantly shorter session lengths and durations, slower typing speeds, and higher error rates.
  • Frequency-domain analysis showed significant differences in the number of peaks and major amplitudes between different survey moments.
  • The machine learning model achieved an average F-score of 93% in cross-validation, validating the effectiveness of time-domain and frequency-domain features in identifying suitable survey moments.

• Advantages over Existing Solutions:

  • Improved adaptability of ESM question timing, enhancing user experience and data quality;
  • More sensitively captures hidden emotional features in keyboard behavior, offering a more innovative approach compared to traditional context-based methods.

• Experiments and Evaluation Results:

  • Based on data from 22 participants over a 3-week experiment, including 3,463 keyboard sessions (83.3% identified as suitable moments), the model demonstrated performance surpassing the baseline (using single features only).
  • SHAP analysis indicated that session length (time-domain) and the first peak amplitude (frequency-domain) were the most influential features in the model.

• Limitations and Future Directions:

  • The sample consisted mainly of students, which may limit generalizability;
  • Future work could expand the sample size and emotional dimensions to further improve model performance;
  • Explore the integration of keyboard interaction data with other sensor data (e.g., accelerometer or voice data) to enhance the precision of emotion sensing.