Deepfake, Real Harm: A Participatory Approach for Imagining Infrastructures to Combat Deepfake Sexual Abuse

Paper Title

Deepfake, Real Harm: A Participatory Approach for Imagining Infrastructures to Combat Deepfake Sexual Abuse

Publication Info

• Topic area: Addressing deepfake sexual abuse through participatory design and survivor-centered interventions.
• Keywords: Deepfake sexual abuse, generative AI, participatory design, survivor-centered design, digital sexual violence, platform accountability, cross-platform coordination, trauma-informed design, AI governance, South Korea.

Background and Problem

• Problem / challenge: Existing systems for addressing non-consensual intimate imagery (NCII) are reactive, platform-centric, and poorly aligned with the workflows of activists and survivor support professionals. They fail to prevent harm, ensure cross-platform coordination, or address the psychological toll on those combating deepfake sexual abuse.
• Significance: Deepfake sexual abuse is a rapidly escalating global issue, with profound psychological, social, and legal implications for victims and activists. The ease of creating and distributing such content exacerbates harm and undermines trust in digital spaces.
• Motivation and related work: Prior efforts have focused on post-hoc detection, watermarking, and platform-specific moderation, which are insufficient for addressing the scale and complexity of deepfake abuse. This study builds on participatory and speculative design approaches to co-develop survivor-centered interventions and address systemic gaps.

Solution

• Proposed approach: A participatory design framework involving activists and survivors to co-imagine technological and policy interventions for combating deepfake sexual abuse.
• Novelty:
  1. Detailed identification of challenges faced by activists and survivors in addressing deepfake sexual abuse.
  2. Design implications for cross-platform coordination, proactive safeguards, and trauma-responsive systems.
  3. Introduction of a participatory design methodology that integrates speculative design to foreground lived experiences.
• Procedure and key techniques:
  • Conducted participatory design workshops with 10 South Korean activists and survivors.
  • Used journey mapping, problem definition, and speculative design activities to identify challenges and co-create solutions.
  • Applied the Value Sensitive Design framework to align interventions with ethical and social values.

Results

• Concrete findings:
  • Deepfake sexual abuse introduces unique challenges, including the ease of perpetration, difficulty in identifying perpetrators, and the psychological harm caused by the mere possibility of victimization.
  • Current platform responses are fragmented and place undue burdens on victims to report and prove harm.
  • Activists face significant emotional and psychological harm due to prolonged exposure to abusive content.
• Advantage over baselines:
  • Proposed interventions focus on prevention, cross-platform coordination, and reducing the burden on survivors and activists, addressing gaps in existing reactive and platform-specific systems.
• Experiments / evaluation:
  • Workshops revealed systemic challenges and co-designed interventions, such as integrated case management systems, cross-platform reporting, and trauma-informed monitoring tools.
  • Participants proposed proactive safeguards, including input restrictions on generative AI and international collaborative investigation systems.
• Limitations and future work:
  • Findings are specific to the South Korean context and may not generalize globally.
  • Lack of participation from police investigators limits insights into law enforcement perspectives.
  • Speculative design outputs may not be immediately technically feasible, requiring further refinement and cross-sector collaboration.

Summary

This study explores the challenges of deepfake sexual abuse and proposes survivor-centered interventions through participatory design workshops with South Korean activists and survivors. Key contributions include proactive safeguards, cross-platform coordination mechanisms, and trauma-informed monitoring tools. The findings highlight the need for structural support for activists, international collaboration, and a shift in platform accountability to reduce harm and support recovery. While grounded in the South Korean context, the insights have broader implications for addressing deepfake sexual abuse globally.