PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

Title of the Paper

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

Paper Information

• Research Domain: Human-AI collaboration, audio-visual data annotation, AI-assisted tools
• Keywords: Human-AI collaboration, data annotation, data labeling, audio-visual learning, interactive machine learning

Research Background and Issues

• What problems or challenges did the authors identify?

  • Audio-visual learning requires high-quality, large-scale datasets, but the lack of such datasets limits model performance and adoption.
  • Annotating audio-visual data is time-consuming, labor-intensive, and repetitive. Existing tools like VIA exhibit inefficiencies in workflows requiring full manual involvement.
  • Many existing annotation tools focus primarily on single-modal data (e.g., images or text) and lack the capability to handle dual-modal data like audio-visual content.

• Why is this problem important?

  • Audio-visual learning is widely applied in fields such as video retrieval, augmented reality (AR/VR), and accessibility design.
  • High-quality datasets are critical for improving the performance of audio-visual models.
  • Reducing annotation costs and enhancing efficiency will promote the adoption of related technologies.

• Motivation and Related Work

  • Existing research has proposed AI-assisted annotation tools, but most are limited to single-modal data.
  • Human-AI collaboration in data science workflows has garnered significant attention, especially in complex tasks involving AI-assisted practices.
  • Educational background and annotation experience may influence tool usage efficiency, which warrants further exploration.

Solution

• What methods or solutions did the authors propose?

  • Developed an audio-visual data annotation tool called "Peanut," which reduces manual annotation workload through human-AI collaboration.
  • Peanut employs single-modal models to partially automate tasks, processing audio and visual features separately, and completing cross-modal annotations with user assistance.
  • Utilizes active learning to improve model performance in real-time, leveraging user annotations to fine-tune the visual-acoustic mapping model.

• What are the innovative aspects of this solution?

  • Peanut introduces a novel multi-modal interaction strategy, combining single-modal model results with user feedback to accomplish annotation tasks, offering partial automation without fully relying on AI.
  • Dynamically recommends keyframes for annotation, significantly reducing the need for users to annotate frame-by-frame.
  • Implements real-time active learning to adapt to scene and content changes in videos.
  • Provides a rapid review feature for annotation results, minimizing users' dependence on AI-generated outcomes.

• What are the implementation steps and key technologies used?

  • Peanut's design uses a binary search algorithm to recommend keyframes.
  • Audio labeling models and object detection models predict annotations for individual frames.
  • Active learning strategies optimize model performance based on user input in real-time.
  • Features thumbnail browsing and annotated video playback to allow users to verify annotation accuracy.

Research Outcomes

• What specific outcomes were achieved?

  • Peanut significantly improved the efficiency of audio-visual data annotation, with users annotating a substantially higher number of frames.
  • In experiments, annotation accuracy (cIoU) increased from 0.72 with manual annotation to 0.93 using Peanut, far surpassing the 0.33 accuracy of fully automated models.

• What advantages does it have compared to existing solutions?

  • Human-AI collaborative tools are better suited for multi-modal data annotation, dynamically adjusting workflows to accommodate video content changes.
  • Peanut's keyframe recommendation algorithm reduces manual annotation workload while maintaining high-quality results.
  • Peanut integrates user control with AI efficiency, even for users without specialized skills or expertise.

• What were the experimental or evaluation results?

  • Experimental participants annotated an average of 488.85 frames in 25 minutes using Peanut, compared to 169.45 frames under baseline conditions.
  • Annotation accuracy was significantly improved with Peanut, and user experience and acceptance were largely positive, particularly regarding time savings and reduced repetitive tasks.

• Limitations and Future Directions

  • Peanut currently does not integrate speech and natural language processing models, which may limit its effectiveness in annotating human speech content.
  • The current version is restricted to sound object localization tasks but could be expanded to event localization or audio-visual parsing tasks.
  • Future plans include public deployment of the tool and its application in large-scale dataset annotation to further validate its effectiveness.