Collab: Fostering Critical Identification of Deepfake Videos on Social Media via Synergistic Annotation

Paper Title

Collab: Fostering Critical Identification of Deepfake Videos on Social Media via Synergistic Annotation

Publication Info

• Topic area: Collaborative tools for deepfake video identification on social media.
• Keywords: Deepfake detection, collaborative annotation, spatio-temporal labeling, social influence, collective intelligence, misinformation, user interface design, video analysis, crowdsourcing, critical thinking.

Background and Problem

• Problem / challenge: Existing tools for identifying deepfake videos are inadequate for spatio-temporal annotation, lack robust aggregation mechanisms, and fail to present insights in ways that enhance user evaluation while mitigating social biases.
• Significance: Deepfake videos pose a significant threat to public discourse, requiring scalable, user-friendly tools to empower critical evaluation and mitigate misinformation.
• Motivation and related work: Prior approaches include automated detection systems, human moderation, and crowdsourced annotations. However, these methods struggle with scalability, brittleness to novel manipulations, and lack interfaces tailored for video-specific challenges. This paper builds on collective intelligence (CI) and social influence (SI) theories to address these gaps.

Solution

• Proposed approach: Collab, a web plugin for collaborative annotation of deepfake videos, integrating spatio-temporal labeling, confidence-weighted aggregation, and hierarchical demonstration.
• Novelty:
  1. A confidence-weighted spatio-temporal Intersection-over-Union (IoU) algorithm for aggregating user annotations.
  2. An intuitive annotation interface for spatio-temporal labeling with confidence scores and rationales.
  3. A hierarchical visualization strategy to present aggregated insights while minimizing social conformity bias.
• Procedure and key techniques:
  • Users annotate videos by marking spatio-temporal regions, assigning labels, confidence scores, and optional rationales.
  • A confidence-weighted 3D IoU algorithm aggregates annotations into consolidated regions and labels.
  • Aggregated insights are displayed as semi-transparent overlays with hierarchical details to guide user evaluation.

Results

• Concrete findings:
  • Collab achieved an F1-score of 0.883, outperforming the No Label (0.795) and No Agg (0.849) conditions.
  • Users annotated an average of 190.8 videos with Collab, compared to 62.1 in the No Label condition.
  • Collab annotations were more precise (median bounding box area = 1.94%) and specific (e.g., increased use of labels like "Blurry").
• Advantage over baselines:
  • Collab improved accuracy by 8.8% over No Label and 3.4% over No Agg.
  • Encouraged critical engagement, reduced cognitive load, and fostered reflective thinking compared to baselines.
• Experiments / evaluation:
  • A 7-day study with 90 participants using a simulated social media platform.
  • Videos from four datasets (Face Forensics++, BioDeepAV, DFW, DDL) were used, with 240 videos (120 real, 120 fake).
  • Metrics included accuracy, bounding box precision, label distribution, and subjective ratings (e.g., NASA-TLX).
• Limitations and future work:
  • Simulated environment with balanced datasets may not fully reflect real-world conditions.
  • Future work should explore deployment in naturalistic settings, integration with automated systems, and adaptations for passive users.

Summary

Collab is a collaborative annotation tool designed to enhance the identification of deepfake videos on social media by integrating spatio-temporal labeling, confidence-weighted aggregation, and hierarchical visualization. A 7-day study demonstrated its effectiveness, achieving an F1-score of 0.883 and fostering critical engagement and precise annotations. Collab outperformed baseline conditions in accuracy, user satisfaction, and annotation quality. Future research should focus on real-world deployment, addressing adversarial manipulation, and expanding the tool's applicability to other media formats.