Beyond the Naked Eye: Empirical Study of How People Perceive, Detect, and Respond to AI-Manipulated Videos

Paper Title

Beyond the Naked Eye: Empirical Study of How People Perceive, Detect, and Respond to AI-Manipulated Videos

Publication Info

• Topic area: Public perception, detection, and response to AI-manipulated videos.
• Keywords: AI-manipulated media, deepfakes, detection accuracy, public perception, verification practices, confidence calibration, social media, misinformation, detection tools, empirical study.

Background and Problem

• Problem / challenge: AI-manipulated videos are increasingly realistic and prevalent, making it difficult for the public to distinguish them from authentic content. Prior studies have focused on niche populations or constrained datasets, leaving gaps in understanding how the general public perceives, detects, and responds to such media.
• Significance: The misuse of AI-manipulated videos poses risks such as political misinformation, fraud, and erosion of trust in digital content. Understanding public engagement with this issue is critical for designing effective interventions and tools.
• Motivation and related work: Previous research has examined specific communities' reactions, journalistic discourse, and human detection ability in lab settings. However, these studies lack ecological validity and fail to address broader public behaviors and perceptions. This paper aims to fill these gaps by studying a representative U.S. sample using real-world video stimuli.

Solution

• Proposed approach: A three-part survey study with 490 U.S. participants to examine perceptions, detection accuracy, confidence calibration, and verification behaviors regarding AI-manipulated videos.
• Novelty:
  1. Use of a diverse, in-the-wild video corpus to improve ecological validity.
  2. Examination of demographic and behavioral factors influencing perceptions and detection accuracy.
  3. Analysis of public awareness and usage of AI detection tools.
  4. Holistic integration of perception, detection, and response behaviors in a unified framework.
• Procedure and key techniques:
  1. Surveyed participants on demographics, media habits, and perceptions of AI-manipulated media prevalence.
  2. Tested detection ability using 13 videos (authentic and manipulated) and collected confidence levels and reasoning.
  3. Analyzed self-reported cues, trust behaviors, and verification actions, including awareness of detection tools.

Results

• Concrete findings:
  • Mean detection accuracy was 66.3%, with higher success for authentic videos (75.6%) than manipulated ones (59.4%).
  • Confidence was poorly calibrated: 22.4% were overconfident, and 27.6% were underconfident.
  • Participants rated audio/video synchronization (mean: 4.47/5) and facial expressions (4.38/5) as the most influential cues for suspicion.
  • 91.5% of participants were unaware of detection tools, and only 8.5% knew of such tools, with even fewer using them.
• Advantage over baselines: The study provides a more ecologically valid understanding of public engagement with AI-manipulated videos compared to prior work relying on lab-curated datasets or niche populations.
• Experiments / evaluation:
  • Participants evaluated 13 videos varying in resolution, content, and manipulation type.
  • Demographic and behavioral predictors of perceptions and accuracy were analyzed.
  • Verification behaviors and tool awareness were assessed through multiple-choice and open-ended questions.
• Limitations and future work:
  • Limited control over participants' viewing environments and potential biases from self-reporting.
  • U.S.-centric sample limits generalizability; cross-cultural studies are needed.
  • Results are a snapshot in time due to the evolving nature of AI technologies.

Summary

This study investigates how the general public perceives, detects, and responds to AI-manipulated videos using a representative U.S. sample and real-world video stimuli. Findings reveal modest detection accuracy, poorly calibrated confidence, and reliance on human-centric cues like facial expressions and audio synchronization. Despite significant research on detection tools, public awareness and usage remain low. The study highlights the need for sociotechnical interventions to enhance public resilience against manipulated media, including education, improved tools, and platform-level solutions. Future work should explore cross-cultural contexts and qualitative insights into public behaviors.