OmniScribe: Authoring Immersive Audio Descriptions for 360° Videos

Title of the Paper

OmniScribe: Authoring Immersive Audio Descriptions for 360° Videos

Paper Information

• Authors: Ruei-Che Chang, Liang-Jin Chen, Chao-Hsien Ting, Yu-Tzu Chao, Chia-Sheng Hung, Bing-Yu Chen, Wan-Chen Lee, Anhong Guo
• Conference: UIST '22, Bend, OR, USA
• Date: October 29 - November 2, 2022
• DOI: https://doi.org/10.1145/3526113.3545613
• Research Area: Accessibility Technology and Human-Computer Interaction
• Keywords: 360° video, audio description, virtual reality, multimedia, accessibility technology, visual impairment, sound visualization, computer vision, mobile applications

Research Background and Problem Statement

• Identified Problems:

  • Video accessibility for blind and visually impaired (BVI) individuals primarily relies on audio descriptions (AD). However, the complexity of omnidirectional visual content and spatial information in 360° videos makes authoring audio descriptions more challenging.
  • Current standard audio description methods fail to capture the immersive advantages of 360° videos, potentially preventing blind users from experiencing a similar level of engagement as sighted users.
  • Describers face challenges with 360° content, including difficulty in panoramic observation, prioritizing multiple focal areas, and time constraints.

• Significance:

  • 360° videos represent a significant future medium for immersive content delivery. Improving audio descriptions can help bridge the digital divide between blind and sighted users.
  • Enhancing the immersive quality of audio descriptions not only improves the video experience for blind users but also advances the application of immersive media in the accessibility technology domain.

• Motivation and Related Work:

  • Traditional audio description tools are primarily designed for 2D videos and standard viewing experiences, making them unsuitable for 360° content.
  • Existing research highlights the potential of spatial audio and interactive methods to enhance 360° audio descriptions but lacks concrete design frameworks and tools, as well as a comprehensive understanding of the needs of describers and blind users.

Proposed Solution

• Proposed Method:

  • Developed OmniScribe, a system for authoring immersive audio descriptions for 360° videos.
  • OmniScribe consists of two core components: a web-based authoring interface for describers and a mobile prototype application for blind users to experience the content.

• Innovations:

  • Provides AI-generated visual content assistance (e.g., region segmentation, saliency overlays, and object tracking) to help describers quickly comprehend 360° video content.
  • Introduces three novel "immersive tags": spatial audio descriptions, scene descriptions, and object descriptions, enhancing the immersive interaction experience for blind users.
  • Designed to accommodate describers with varying levels of technical expertise, ensuring a low barrier to entry.

• Implementation Steps and Key Technologies:

  1. Content Understanding Assistance: Offers equirectangular projections and localized views of panoramic videos; includes region segmentation boundaries, salient object outlines, and object tracking overlays.
  2. Immersive Tag Creation: Enables describers to author spatially relevant audio descriptions for video scenes and generate object descriptions by annotating movement paths and directions.
  3. Multimodal Interaction Design: Allows BVI users to experience haptic cues and directional AD audio through the mobile app, with the ability to explore virtual object locations using mobile devices.
  4. Video Preprocessing Pipeline: Incorporates AI-driven scene segmentation, audio segmentation, saliency detection, and object tracking technologies.

Research Outcomes

• Specific Achievements:

  • Developed the OmniScribe system and successfully implemented its core functionalities.
  • Demonstrated the effectiveness of OmniScribe in the describer’s authoring process and the improved user experience of immersive audio descriptions for blind users through experiments.

• Advantages Comparison:

  • Compared to standard AD tools, OmniScribe significantly improves the efficiency and comprehensiveness of describers working with 360° content, while providing blind users with an immersive interaction experience.
  • Experimental data shows that blind users prefer the immersive AD generated by OmniScribe over standard AD, citing higher information richness.

• Experimental or Evaluation Results:

  • Evaluation with Describers:
    • Participants: 8 novices and 3 professional describers.
    • Results: Novices found OmniScribe's learning curve slightly steep but appreciated its rich functionality; professionals reported that the system significantly enhanced their understanding of spatial information in videos.
  • Evaluation with BVI Users:
    • Participants: 8 blind users (aged 20-32).
    • Results: All users highly appreciated the "object description" feature, and the majority preferred the immersive AD created by OmniScribe, citing its superior interactivity and immersion.

• Limitations and Future Directions:

  • Limitations:
    • Describers face challenges in prioritizing content under time constraints.
    • Providing immersion may increase cognitive load for users.
  • Future Directions:
    • Explore ways to better balance cognitive load and information immersion.
    • Introduce more interactive technologies to enhance mobile interaction, such as touch gestures or advanced audio feedback.
    • Support personalized description generation and presentation based on user interests, and expand applications to other accessible multimedia domains.