Front Row: Automatically Generating Immersive Audio Representations of Tennis Broadcasts for Blind Viewers

Document Title

Front Row: Automatically Generating Immersive Audio Representations of Tennis Broadcasts for Blind Viewers

Document Information

• Domain: Human-Computer Interaction, Computer Vision, Accessibility Design
• Keywords: Visual impairment, sports, accessibility technology, computer vision, immersive audio, blind user experience, sports broadcasting, accessible video

Research Background and Problem

• Identified Challenges:

  • Individuals with visual impairments struggle to fully comprehend sports content through conventional television or radio media; existing text or voice descriptions fail to provide spatial information.
  • Visually impaired individuals rely on explanations from spectators or commentators, which limits their ability to independently understand the match.
  • Current technologies (e.g., high-precision cameras and sensors) rely on expensive hardware and are applicable only to select sports events.

• Significance: Sports broadcasting is an important way for visually impaired individuals to integrate into society and culture, yet its inaccessibility leaves many feeling isolated or excluded.

• Research Motivation and Related Work:

  • Previous studies have explored tactile graphics and sound effect technologies, but these methods are unsuitable for rapidly changing scenes like sports videos.
  • Systems like Action Audio depend on hardware installations at venues, whereas Front Row aims to generate audio representations directly from video, enhancing the technology's universal applicability.

Solution

• Proposed Approach:

  • Front Row System: Extracts player positions and actions from tennis match videos using computer vision and presents match details in spatialized (3D) audio format.
  • The system features three primary audio cues: player positions (continuous low-frequency sound), player actions (bell tones distinguishing forehand and backhand), and ambient game sounds (e.g., audience cheers).

• Innovations:

  • Generates audio representations directly from video, eliminating hardware dependency and enabling accessibility for all sports broadcasts.
  • Unprecedented integration of spatialized audio technology to provide visually impaired users with real-time dynamics and positional understanding.
  • Empowers visually impaired users to independently perceive match content and analyze player strategies.

• Implementation Steps and Key Technologies:

  1. Computer Vision Processing:
     • Detects the court, ball, and player positions using deep learning models to track the ball and players.
     • Identifies hitting moments through trajectory changes and detects hitting types (forehand/backhand) from frames around the player.
     • Uses Support Vector Machines to identify "active match moments" in the video (e.g., ignoring rendering during side changes).
  2. Audio Design:
     • Utilizes Steam Audio tools to create 3D audio scenes, rendering the court as a two-dimensional plane in front of the user.
     • Integrates ambient audio (e.g., audience sounds) with generated sound effects to enhance realism for visually impaired users.
  3. User-Customized Experience:
     • Provides intuitive spatial position cues and adjustable audio modes tailored to users with varying levels of technical expertise.

Research Outcomes

• Specific Results:

  • Front Row significantly improved users' understanding of match content, reducing error rates: for instance, identifying hitting types saw a 90% reduction in error compared to television.
  • Enhanced immersion: users reported that Front Row enabled them to independently analyze matches and form insights into player mindset and strategies.
  • Overall user experience surpassed radio and television, particularly in spatial understanding and accuracy.

• Advantages Comparison:

  • Compared to television: Front Row offers real-time content interpretation capabilities.
  • Compared to radio: Front Row avoids commentator bias, providing more neutral and accurate match information.

• Experiment and Evaluation Results:

  • Technical tests showed an average accuracy of 97.2% in player position tracking.
  • User studies revealed that 8 out of 12 visually impaired participants ranked Front Row as their preferred match-viewing experience.

• Limitations and Future Directions:

  • Limitations: The continuous "low-frequency sound" in Front Row's audio design was perceived by some users as lacking realism; the system does not fully cover complex events (e.g., multiplayer sports).
  • Future Directions:
    • Develop multimodal feedback (e.g., tactile + audio) to convey details like hitting force.
    • Support representation of different player numbers and actions in more complex scenarios (e.g., soccer or basketball).
    • Expand accessibility broadcasting to more sports (e.g., badminton, table tennis).