VideoA11y: Method and Dataset for Accessible Video Description

Research Background and Issues

• Identified Problems and Challenges: Current AI models perform poorly in generating video descriptions for blind or low-vision (BLV) users due to the suboptimal quality of training data annotations. This results in descriptions that fail to fully meet the needs of BLV users. Additionally, human-generated descriptions are often incomplete or contain grammatical, spelling, or semantic errors.
• Significance: The rapid growth of video content has further widened the information gap for BLV individuals. While audio descriptions (AD) can address these needs, their production requires expensive professional teams. This is particularly problematic for user-generated content (e.g., YouTube and TikTok), where audio descriptions are scarce and lag behind demand.
• Research Motivation and Related Work: Previous studies have primarily focused on assisting human describers in generating descriptions, while fully automated approaches have yet to effectively address the challenge of producing detailed video descriptions that adhere to AD guidelines. Furthermore, the limitations of existing datasets hinder the quality and usability of AI-generated descriptions.

Solution

• Proposed Method and Innovations:
  • A novel approach, VideoA11y, is proposed, which combines multimodal large language models (MLLMs) with video accessibility guidelines to design standardized prompts for generating highly clear and accurate video descriptions.
  • Based on this method, a VideoA11y-40K dataset containing 40,000 video clips has been constructed, making it the largest and most comprehensive video description dataset tailored for BLV users.
• Implementation Steps:
  1. Extract and organize 42 AD guidelines from professional audio description resources to guide description generation.
  2. Use MLLMs such as GPT-4 Vision (GPT-4V) to generate video descriptions, integrating keyframe data from videos.
  3. Construct and classify the VideoA11y-40K dataset, categorizing videos into 15 classes, with descriptions generated using standardized prompts for each video.
• Key Technologies Used:
  • Multimodal language models (e.g., GPT-4V and Video-LLaVA) for zero-shot prompt optimization.
  • Video keyframe extraction algorithms (based on local maxima algorithms) to ensure significant changes in videos are captured.
  • Dataset preparation, including video classification and rigorous quality evaluation.

Research Outcomes

• Specific Outcomes:
  1. Descriptions generated by VideoA11y outperform novice human annotations in clarity, accuracy, objectivity, and descriptiveness, approaching the quality of high-quality human annotations.
  2. The constructed VideoA11y-40K dataset significantly improves the quality of descriptions generated by multimodal language models, providing a robust data foundation for video accessibility.
  3. A new benchmark standard is proposed for evaluating video description models designed for BLV users.
• Advantages:
  • VideoA11y-generated descriptions surpass novice human annotations in terms of accuracy and clarity, closely matching professional human annotations.
  • The VideoA11y method significantly enhances existing models while reducing reliance on human involvement, offering high scalability.
  • Reduced hallucination phenomena (i.e., generating inaccurate content), with notable improvements in accuracy when human annotations are used as references.
• Experiments and Evaluation Results:
  • Across five user studies (involving 347 sighted users, 40 BLV users, and 7 professional describers), VideoA11y demonstrated outstanding performance:
    • BLV users preferred VideoA11y-generated descriptions in over 90% of scenarios.
    • Professional describers favored VideoA11y-generated descriptions, considering them aligned with professional standards.
  • Technical experiments showed that open-source MLLM models fine-tuned with the VideoA11y-40K dataset significantly outperformed baseline models without fine-tuning.
• Limitations and Future Directions:
  • Limitations:
    1. In the absence of human annotations, minor hallucination phenomena may occur (e.g., introducing details not present in the video).
    2. The current method cannot fully meet BLV users' personalized needs (e.g., preferences for description length or level of detail).
    3. Seamless integration of descriptions with video content (e.g., inserting natural pause points) has not been fully achieved.
  • Future Directions:
    1. Optimize generation models to reduce hallucination scenarios, for example, by employing direct preference optimization (DPO) techniques or integrating auxiliary models to enhance input accuracy.
    2. Collect BLV user preference data to enable dynamic adjustment of description style and level of detail.
    3. Integrate with existing systems (e.g., Rescribe) to support embedded descriptions and seamless presentation within video streams.