EmotiV: Exploring Automatic Emotion Sharing through Facial Expression Recognition (FER) for Online Co-Watching

Paper Title

EmotiV: Exploring Automatic Emotion Sharing through Facial Expression Recognition (FER) for Online Co-Watching

Publication Info

• Topic area: Enhancing remote video co-watching using emotion recognition.
• Keywords: Facial Expression Recognition, FER, co-watching, emotion sharing, online video, social presence, self-awareness, privacy, user experience.

Background and Problem

• Problem / challenge: Existing co-watching technologies often require active user input (e.g., typing) or specialized devices, which can interrupt the viewing experience or limit accessibility. There is a lack of lightweight, non-intrusive solutions for fostering social presence during remote video watching.
• Significance: Enhancing remote co-watching can improve emotional alignment, social closeness, and the overall viewing experience, making solitary media consumption more engaging and communal.
• Motivation and related work: Prior work includes text-based systems like Danmaku, synchronized playback platforms, VR-based co-watching, and bio-signal sensing, but these approaches either disrupt the flow of watching or require specialized hardware. Facial Expression Recognition (FER) has been explored in other domains but remains underutilized for co-watching.

Solution

• Proposed approach: EmotiV, a web-based prototype leveraging FER to automatically capture and share viewers’ emotions using standard webcams, providing real-time emotional feedback through visual overlays.
• Novelty:
  1. Lightweight, non-intrusive emotion sharing using FER without requiring active user input or additional devices.
  2. Two co-watching modes: Emotion Bubble (sidebar visualization) and Emotion Danmaku (emoji reactions drifting across the video).
  3. Integration of real-time and post-watch feedback for self-awareness and reflection.
• Procedure and key techniques:
  • FER model (ResEmoteNet) trained on public datasets and refined with webcam-collected data for 84% accuracy.
  • Real-time emotion detection using a sliding-window voting mechanism for stability.
  • Two visualization modes: Emotion Bubble (aggregated emotions in a sidebar) and Emotion Danmaku (real-time emoji overlays).
  • Post-watch Emotion Timeline Review for reflection on emotional reactions.

Results

• Concrete findings:
  • Participants reported heightened enjoyment (12 agreed, 8 strongly agreed for Emotion Bubble; 14 agreed, 6 strongly agreed for Emotion Danmaku).
  • EmotiV evoked a sense of "alone watching together," amplifying fun, companionship, and emotional contagion.
  • Self-awareness was promoted through real-time feedback and post-watch reflection.
• Advantage over baselines: Compared to text-based bullet comments, EmotiV provided more timely and authentic emotional feedback without requiring active input, though it offered less expressive richness and control.
• Experiments / evaluation:
  • User study with 20 participants watching a 15-minute Mr. Bean clip in three segments: baseline (no feedback), Emotion Bubble, and Emotion Danmaku.
  • Data collected through structured questionnaires and semi-structured interviews.
  • Thematic analysis revealed themes of togetherness, self-awareness, and privacy concerns.
• Limitations and future work:
  • Limited to a single comedy video; future work should explore diverse genres (e.g., horror, sports).
  • Participant sample was from a single university; broader demographics are needed.
  • Conducted in a controlled lab environment; naturalistic settings should be studied.
  • Long-term effects and sustainability of FER-based co-watching remain unexplored.

Summary

EmotiV is a FER-based system that enhances remote video co-watching by automatically capturing and sharing viewers’ emotions through lightweight visual overlays. It fosters a sense of "alone watching together," amplifies fun and companionship, and promotes self-awareness of emotional reactions. While it offers timely and authentic feedback, it lacks the richness of text-based interactions and raises privacy concerns. Future work should explore diverse genres, broader participant samples, and long-term effects to refine and expand its applicability. EmotiV provides valuable insights into designing affect-aware systems for connected and reflective media consumption.