Sweating the Details: Emotion Recognition and the Influence of Physical Exertion in Virtual Reality Exergaming

Title of the Paper

Sweating the Details: Emotion Recognition and the Influence of Physical Exertion in Virtual Reality Exergaming

Paper Information

• Field of Study: Research on emotion recognition and the impact of physical activity on emotions in virtual reality
• Keywords: Virtual reality, exergaming, emotion recognition, physiological sensors, emotional impact, high-intensity exercise, user experience

Research Background and Issues

• Identified Problems and Challenges:

  • Recognizing users' emotional states in virtual reality (VR) gaming has significant potential for adaptive game design. However, physical activity and VR experiences interfere with physiological sensors, making emotion recognition challenging.
  • Current research focuses on emotion recognition during moderate-intensity exercise or in non-VR environments, with limited studies addressing high-intensity exercise or multidimensional emotional states.
  • Physiological sensor data is affected by movement, individual differences, and VR environmental factors, necessitating robust data cleaning methods.

• Significance:

  • Understanding the dynamic changes in users' emotional states during VR gaming can optimize user experience and enhance game engagement.
  • Efficient emotion recognition technologies can support real-time emotional interaction within game environments, making games more immersive and optimized.

• Motivation and Related Work:

  • The authors designed a series of experiments to study the relationship between physical activity and users' emotional states and validated emotion-inducing environmental designs.
  • The background involves empirical studies on emotion models (e.g., Russell's Circumplex Model of Affect) and the application of physiological sensors and psychophysiological data, such as pupil activity, skin conductance, heart rate variability, and facial expressions.

Solution

• Proposed Approach:

  • Develop four virtual environments (targeting emotions of happiness, calmness, stress, and sadness) and design a VR cycling game using the Unity engine for experiments.
  • Collect various physiological data (pupil activity, skin conductance, heart rate, facial tracking) and subjective emotional ratings from users.
  • Apply multi-level linear regression models to test methods for predicting emotional states using physiological data and compare the effects of different levels of data cleaning (raw, environment-cleaned, and personalized-cleaned).

• Innovations:

  • Conduct a systematic study on the impact of physical activity intensity on emotion recognition, covering low, medium, and high intensity levels for the first time.
  • Propose validated emotion-inducing virtual environments and cleaning methods to eliminate the influence of environmental and individual differences on physiological data.
  • Provide an open dataset and analysis framework (EmoSense SDK) to support other researchers in physiological data collection and analysis.

• Implementation Steps and Techniques:

  1. Validate whether the designed virtual environments can induce target emotions and eliminate confounding factors.
  2. Collect physiological and psychological data from 72 participants through experiments using a balanced Latin square design.
  3. Apply linear regression models to analyze the effectiveness of physiological data in predicting emotions.
  4. Compare the impact of three cleaning methods on model prediction performance and consistency.

Research Findings

• Specific Results:

  • Successfully validated the design of emotion-inducing virtual environments, confirming through statistical and experimental results that target emotions can be elicited.
  • Proposed a series of regression models capable of predicting multiple emotional dimensions (e.g., valence, arousal), with pupil dilation (PDL/PDR) and cycling power output being the strongest predictors.
  • Cleaning physiological sensor data significantly improved the accuracy of emotion recognition predictions and the fit of the models.

• Advantages Over Existing Solutions:

  • Compared to existing studies that focus only on single intensity levels or simple emotion models, the authors' work covers a broader spectrum of emotions and variations in exercise intensity.
  • The proposed cleaning methods (personalized processing) significantly improved model prediction performance and reduced the likelihood of violating regression model assumptions.

• Experimental or Evaluation Results:

  • Higher levels of data cleaning led to stronger model predictive power, greater consistency, and significant reduction in environmental interference factors.
  • In emotion recognition, pupil dilation and power output were the strongest predictors of emotional states.

• Limitations and Future Directions:

  • The study only validated emotional state changes during cycling as a form of exercise, requiring further exploration of other exercise forms (e.g., running, weightlifting).
  • Future work could adopt machine learning methods to enhance emotion recognition models while considering complex game interactions and social contexts.
  • Investigate the longitudinal effects of repeated gaming experiences to further evaluate the dynamic changes in emotional states.

Experimental Data and Tools

• An open real-time dataset includes physiological sensor measurements and user emotional ratings across four virtual environments.
• Provides the open-source tool EmoSense SDK for real-time physiological data collection and analysis.

Practical Guidance

• Data Cleaning: Remove environmental and individual factors, and improve prediction accuracy through personalized standardization.
• Physiological Sensor Selection: Ensure high-quality pupil tracking and logical calibration, prioritizing the collection of cycling power output and skin conductance.
• Model Application: Use linear regression models to predict emotional states, while integrating game design to select appropriate exercise intensities for targeted emotions.

This study combines psychological models with computer science technologies, advancing the field of emotional interaction in virtual reality and offering new insights and tools for emotion recognition in high-intensity exercise environments.