RCEA-360VR: Real-time, Continuous Emotion Annotation in 360° VR Videos for Collecting Precise Viewport-dependent Ground Truth Labels

Title of the Paper

RCEA-360VR: Real-time, Continuous Emotion Annotation in 360° VR Videos for Collecting Precise Viewport-dependent Ground Truth Labels

Paper Information

• Field of Study: Affective Computing and Human-Computer Interaction in Virtual Reality
• Keywords: Emotion, Annotation, 360° Video, Virtual Reality, Labels, Real-time, Continuous, Viewport-dependent, Head Movement

Research Background and Problem

• Problem or Challenge: Generating precise emotion labels in 360° virtual reality (VR) environments is challenging due to the diverse ways users interact with video content. Traditional emotion annotation methods fail to capture real-time dynamic emotional changes in virtual environments, while retrospective self-reports are prone to memory bias. Additionally, user-adjustable viewports may lead to inconsistencies between emotion annotations and specific scenes.
• Significance: Accurate emotion labels are critical for emotion recognition systems and machine learning models. They have broad applications in fields such as education, virtual tourism, and news engagement, while also supporting emotion analysis in immersive VR environments.
• Research Motivation and Related Work:
  • Traditional emotion annotation methods (e.g., SAM self-assessment questionnaires) cannot capture real-time emotional dynamics during video viewing.
  • Existing studies have developed real-time emotion annotation techniques for desktop and mobile platforms, but solutions tailored to 360° VR environments are lacking.
  • Literature reviews indicate that emotion annotation based on viewport and head movement has not been thoroughly explored.

Solution

Method & System Design

• Proposed Solution: Development of the RCEA-360VR system, incorporating two peripheral visualization techniques, HaloLight and DotSize, to assist users in real-time, continuous emotion annotation in 360° VR videos.
• Innovations:
  • Achieving accurate viewport-dependent emotion labels by recording users' head movement dynamics for emotion integration.
  • Introducing a segment-level viewport clustering algorithm based on user distribution to generate precise emotion labels.
  • Employing peripheral feedback design to reduce users' cognitive and visual load without disrupting VR immersion.
• Implementation Steps:
  1. Behavioral Data Collection: Using HTC Vive Pro Eye to record head movement and eye-tracking data, with Joy-Con controllers for emotion annotation.
  2. Real-time Emotion Annotation Design:
     • HaloLight: Displays a transparent halo in the bottom-right corner of the viewport to indicate annotation status and emotion intensity.
     • DotSize: Displays a dot in the bottom-right corner of the viewport, with its size representing emotion intensity.
  3. Fusion Algorithm:
     • Temporal Alignment: Uses Dynamic Time Warping (DTW) to calibrate time delays in annotation sequences.
     • Viewport Clustering: Generates viewport clusters based on head movement data to filter out major viewport user groups.
     • Annotation Fusion: Employs Bayesian fusion combined with a segmentation framework to aggregate frame-by-frame emotion labels into final ground truth emotion labels.

Research Outcomes

• Specific Results:

  • The RCEA-360VR system can collect fine-grained, viewport-dependent emotion labels.
  • HaloLight and DotSize techniques demonstrated high usability, with no significant increase in cognitive load, motion sickness, or disruption of immersion during experiments.
  • Experimental results showed high consistency between emotion labels generated by the two techniques, discrete emotion evaluation methods (e.g., SAM questionnaires), and original video labels.
  • Emotion labels generated through the fusion algorithm achieved 100% classification accuracy in validation experiments.
  • Provided more detailed temporal emotion state analysis, capturing emotional fluctuations at specific viewpoints.

• Comparison with Existing Solutions:

  • Compared to static 2D grids or retrospective emotion reports, the real-time, continuous visual annotation method of RCEA-360VR significantly improves the accuracy of viewport-dependent labels.
  • HaloLight and DotSize did not significantly increase user burden and are better suited for the dynamic viewing demands of 360° videos.

• Experimental and Evaluation Results:

  • Subjective Evaluation: NASA-TLX, IPQ, and SSQ questionnaires were used to verify that the visual annotation techniques had minimal impact on users.
  • Objective Evaluation: Physiological data (pupil dilation, EDA changes, heart rate intervals) confirmed that the annotation task did not significantly increase cognitive load.

• Limitations and Future Directions:

  • Limited to short videos (<1 minute); the impact of longer videos on users has not been validated.
  • Currently tested only in 360° video environments, not extended to more complex VR interaction scenarios (e.g., free walking, virtual world operations).
  • Potential associations between eye-tracking data and emotions have not been explored.
  • Design of auxiliary devices and color systems needs optimization for users with visual impairments (e.g., color blindness).

Conclusion

The RCEA-360VR system demonstrates the potential for real-time emotion annotation in 360° VR environments. By introducing viewport-dependent features and emotion fusion algorithms, it enhances the precision and applicability of affective computing research, particularly for generating efficient labels for emotion recognition, prediction, and machine learning model training. Future work should further explore emotion collection and analysis in long-duration, multi-interaction dynamic VR experiences, addressing issues such as user consistency, entry points, and personalized annotations.