RageSense: Leveraging Acoustic Sensing and LLM-Based Intervention for Emotion Regulation in Mobile Gaming

Paper Title

RageSense: Leveraging Acoustic Sensing and LLM-Based Intervention for Emotion Regulation in Mobile Gaming

Publication Info

• Topic area: Emotion detection and regulation in mobile gaming using acoustic sensing and AI-driven interventions.
• Keywords: Emotion recognition, valence-arousal, acoustic sensing, mobile gaming, large language models, intervention systems, ultrasonic sensing, gameplay experience, emotion regulation, privacy-friendly technology.

Background and Problem

• Problem / challenge: Existing emotion recognition methods rely on intrusive technologies like cameras or wearables, face challenges in dynamic gaming environments, and fail to provide timely, personalized interventions for negative emotions such as rage or frustration.
• Significance: Unregulated negative emotions in gaming can lead to problematic behaviors, including compulsive play, gaming addiction, and cyberbullying, highlighting the need for effective emotion regulation systems.
• Motivation and related work: Prior research has explored physiological and behavioral emotion recognition, but these approaches face privacy concerns, environmental limitations, and lack real-time adaptability. Acoustic sensing has shown promise but remains underexplored for continuous valence-arousal dynamics in gaming contexts.

Solution

• Proposed approach: RageSense, a system combining near-ultrasonic acoustic sensing with large language model (LLM)-powered interventions to detect and regulate player emotions during mobile gaming.
• Novelty:
  1. Continuous valence-arousal estimation using acoustic spectrograms enhanced by IMU-informed filtering to suppress motion artifacts.
  2. Personalized, context-aware interventions generated by LLMs based on real-time game content and emotional state.
  3. Comprehensive in-field evaluation demonstrating effectiveness in emotion regulation during gameplay.
• Procedure and key techniques:
  • Acoustic sensing captures facial muscle movements via ultrasonic signals emitted and received by smartphone speakers and microphones.
  • IMU data filters motion-induced noise from hand gestures during gameplay.
  • A hybrid Sparse Transformer + LSTM model maps acoustic features to valence-arousal space.
  • LLM generates empathetic, context-sensitive intervention messages using gameplay screenshots and emotional trajectories.
  • Interventions are delivered as unobtrusive on-screen overlays.

Results

• Concrete findings:
  • RageSense achieved a concordance correlation coefficient (CCC) of 0.6115 for valence and 0.6215 for arousal, with a combined PVA of 0.6165, outperforming other models.
  • Intervention latency ranged from 864 ms to 928 ms across tested smartphones, ensuring real-time feedback.
  • Battery overhead was minimal, adding only 1.60% drain during gameplay.
• Advantage over baselines:
  • RageSense interventions significantly improved valence and reduced arousal compared to predefined text-based and random interventions.
  • Personalized interventions were rated higher in user satisfaction, contextual relevance, and timing appropriateness.
• Experiments / evaluation:
  • Study 1 validated the accuracy of acoustic-based emotion detection using 20 participants and multiple game types.
  • Study 2 involved 53 participants in a three-week field study comparing intervention methods, showing superior emotional regulation and user preference for personalized interventions.
  • Robustness tests confirmed system stability across noisy environments, diverse user characteristics, and varying smartphone models.
• Limitations and future work:
  • Limited long-term emotional regulation effects observed; future studies should explore extended usage periods.
  • Less effective in fast-paced games like MOBA; alternative intervention modalities (e.g., tactile cues) should be investigated.
  • Privacy risks related to affective data and ultrasonic reflections require further minimization and study.

Summary

RageSense introduces a privacy-friendly, real-time emotion detection and regulation system for mobile gaming, leveraging ultrasonic acoustic sensing and LLM-generated interventions. It achieves high accuracy in valence-arousal estimation and delivers personalized, context-aware emotional support, significantly improving player well-being. Field studies confirm its effectiveness in reducing frustration and enhancing gameplay experience, particularly in slower-paced games. While challenges remain in long-term emotional regulation and fast-paced game contexts, RageSense demonstrates strong potential for practical applications in interactive environments.