Towards Understanding Perceptual Differences between Genuine and Face-Swapped Videos

Document Title

Towards Understanding Perceptual Differences between Genuine and Face-Swapped Videos

Document Information

• Subject Area: Perception Research on Face-Swapped Videos
• Keywords: Video Manipulation, Human Perception, Eye Tracking, Face Swapping, Deepfake Detection, Emotion Recognition
• Conference: CHI Conference on Human Factors in Computing Systems (CHI '21), May 2021
• DOI: https://doi.org/10.1145/3411764.3445627

Research Background and Problem

• What problems or challenges did the authors identify?
  • With the advancement of deep learning, face-swapping technology has become increasingly powerful. While it is used for creative entertainment, it can also be misused for unethical forgery, posing societal threats.
  • It remains unclear why humans are misled or what cues they use to detect video manipulation.
• Why is this issue important?
  • Face-swapping technology can be misused in political contexts (e.g., spreading misinformation) and social scenarios (e.g., cyberbullying), affecting societal stability. Understanding human perception and developing reliable detection methods are urgent issues to address.
• Research Motivation and Related Work:
  • Existing detection methods focus primarily on image analysis and identifying traces of forgery, with limited exploration of observer feedback.
  • Humans are highly sensitive to faces and excel at facial recognition. Therefore, studying how humans perceive face-swapped videos can provide new insights for improving detection tools.

Solution

• Research Methods:
  • The paper explores how humans detect manipulation in videos through three perceptual experiments (eye tracking, the effect of video duration on detection accuracy, and emotion evaluation).
• Innovations:
  • Introduced eye-tracking data into the study of perception of face-swapped videos and examined its potential for detecting facial manipulation.
  • Investigated differences in emotional transmission between genuine and manipulated videos, a domain challenging for computational algorithms.
• Implementation Steps:
  1. Experiment Design:
     • Experiment E1 (Eye Tracking): Analyze eye movement behavior when watching genuine and manipulated videos.
     • Experiment E2 (Effect of Video Duration): Test the impact of video length on manipulation detection accuracy.
     • Experiment E3 (Emotion Evaluation): Assess human accuracy, intensity, and sincerity in recognizing emotions in genuine and manipulated videos.
  2. Experimental Data:
     • Used the PEFS dataset and FaceForensics dataset in controlled environments to generate genuine and manipulated video versions.
  3. Data Collection and Analysis:
     • Employed high-speed eye-tracking equipment and online survey platforms to record participants' viewing behavior, video quality reports, and emotion evaluations.

Research Findings

• Specific Findings:
  1. Eye-Tracking Data Analysis:
     • Compared to genuine videos, participants focused more on the mouth and nose and less on the eyes when watching high-quality manipulated videos.
     • While artifacts along facial contours were reported less frequently, they significantly influenced gaze behavior.
  2. Effect of Video Duration on Detection Accuracy:
     • Video duration had no significant impact on the detection accuracy of both genuine and manipulated videos.
     • Results indicate that participants often made their judgments early in the video.
  3. Emotion Recognition:
     • Although overall emotion recognition accuracy was similar, genuine videos slightly outperformed manipulated videos in terms of intensity and sincerity.
     • Certain emotions (e.g., disgust, surprise) were less effectively conveyed in manipulated videos.
• Advantages over Existing Solutions:
  • Introduced observer-related perceptual data, offering a new perspective for existing facial manipulation detection algorithms.
  • Does not directly rely on visual artifacts of forgery, potentially making it robust against more advanced facial manipulation techniques in the future.
• Limitations and Future Directions:
  • Experiments were conducted only in controlled environments, lacking validation in more complex scenarios (e.g., free motion, multi-person scenes).
  • Participants were primarily young university students, which may not fully represent the perception patterns of a broader population.
  • Future research could expand to different types of facial manipulation techniques and explore the influence of familiarity on perception.