Toward Automatic Audio Description Generation for Accessible Videos

Document Title

Toward Automatic Audio Description Generation for Accessible Videos

Document Information

• Subject Area: Accessibility Technology, Computer Vision, and Natural Language Processing
• Keywords: Audio Description, Video Description, Audio-Visual Consistency, Video Subtitles, Sentence-Level Embedding, Accessible Design

Research Background and Problem Statement

• Problems and Challenges:

  1. The current coverage of audio descriptions (AD) in video content is low, especially in user-generated videos, where audio descriptions are almost nonexistent, posing barriers for visually impaired users to consume content.
  2. Manually generating audio descriptions is time-consuming and labor-intensive, lacking scalability.
  3. Generating natural and high-quality audio descriptions faces multiple challenges: when to insert descriptions (When), what content to describe (What), and how to organize the descriptions (How) are not clearly defined.

• Significance of the Research: Audio descriptions not only enhance visually impaired users' understanding of video content but also improve the video experience for general users, such as aiding memory of visual details and supporting language learning.

• Motivation and Related Work:

  1. Current research primarily focuses on manually generating audio descriptions or involves some form of human intervention, with limited work on fully automated generation.
  2. Although there has been progress in audio-visual consistency detection and large-scale video description generation, the issue of audio-visual complementarity in complex video content remains insufficiently addressed.

Solution

Method or Solution

• A three-stage automated audio description generation system named Tiresias is proposed, which includes:
  1. Insertion Time Prediction Module: Predicts the time points for adding audio descriptions using an audio-visual consistency detection algorithm.
  2. Audio Description Generation Module: Generates suitable event descriptions based on dense video captioning.
  3. Audio Description Optimization Module: Optimizes the generated descriptions at the sentence level to enhance accuracy and fluency.

Innovations

• Combines advanced computer vision techniques (e.g., audio-visual embedding learning, feature alignment) and natural language processing technologies (e.g., sentence-level embedding, language models) to build an end-to-end automated audio description generation system.
• Proposes a cost function-based description optimization method that comprehensively considers semantic relevance, diversity, and linguistic fluency.

Implementation Steps

1. Extract keyframes from videos and use deep learning networks to perform audio consistency prediction, identifying silent or inconsistent segments requiring descriptions.
2. Generate event descriptions using a dual-network-based video captioning algorithm.
3. Apply a sentence optimization model to filter, rank, and generate descriptions, reducing redundancy and improving grammatical accuracy. Finally, convert the text into speech and insert it into the original video.

Research Outcomes

• Specific Results:

  1. Tiresias performed satisfactorily on a dataset of 500 videos, significantly reducing the need for additional information among visually impaired users.
  2. Qualitative studies revealed that Blind/Visually Impaired (BVI) user feedback indicated a significant improvement in video comprehension compared to videos without descriptions.
  3. In experiments, 70% of BVI users found the results "helpful," and 86.11% believed the generated descriptions reduced their confusion about video content.

• Experimental Evaluation and Comparison:

  1. The overlap rate between the automatically generated audio description insertion points and user-desired time points was 65.50%, closely aligning with user expectations.
  2. Subjective perceptions of redundancy and accuracy in the generated descriptions varied significantly between different user groups (sighted users and visually impaired users), providing insights for model improvement.

• Limitations and Future Directions:

  1. The current model primarily addresses relatively simple activity recognition tasks and lacks depth in contextual understanding (e.g., causal relationships, character detail descriptions).
  2. Future work should focus on developing dynamic audio description systems that allow users to flexibly adjust the level of detail in descriptions to meet personalized needs.
  3. Enhancing the video understanding module's multimodal learning capabilities to generate semantically richer descriptions that better align with user expectations.

Output Format

In the output report, it is recommended to quantify the effectiveness of audio description generation based on user type and video type. Additionally, a horizontal comparison with existing technologies (e.g., manually generated audio descriptions) can be included to highlight the uniqueness of the proposed solution.