A Systematic Review of User Experiments on the Effects of Dark Patterns

Paper Title

A Systematic Review of User Experiments on the Effects of Dark Patterns

Publication Info

• Topic area: The impact of deceptive/manipulative patterns (DMPs) on user behavior, based on experimental evidence.
• Keywords: dark patterns, deceptive design, manipulative design, user behavior, privacy, consumer harm, interventions, experimental studies, regulatory policy, interface design.

Background and Problem

• Problem / challenge: While dark patterns are widely acknowledged as harmful, empirical evidence quantifying their real-world effects is scattered across disciplines, and the effectiveness of interventions to mitigate their impact remains unclear.
• Significance: Understanding the measurable effects of DMPs is crucial for informing regulatory actions and designing interventions to protect users from manipulative practices.
• Motivation and related work: Prior research has focused on defining, classifying, and describing DMPs, but no comprehensive review has synthesized experimental findings on their behavioral effects. Adjacent reviews have explored nudging, regulatory approaches, and specific subsets of DMPs, leaving a gap in understanding their quantified impacts and the efficacy of countermeasures.

Solution

• Proposed approach: A systematic review of 148 experimental units from 27 peer-reviewed papers (2019–2025) to synthesize evidence on the behavioral effects of DMPs and the effectiveness of interventions.
• Novelty:
  1. Aggregates experimental evidence across all DMP types to assess their behavioral impact.
  2. Evaluates the effectiveness of interventions designed to mitigate DMP effects.
  3. Analyzes the additive effects of multiple DMPs and compares the relative effectiveness of different DMP types.
  4. Investigates correlations between DMP effects and user personal characteristics.
• Procedure and key techniques:
  1. Systematic search across multiple academic repositories (Google Scholar, ACM Digital Library, SSRN, arXiv).
  2. Screening and inclusion criteria: peer-reviewed studies, behavioral measures, statistical significance, and DMP framing.
  3. Classification of experiments into five categories: DMP effects, intervention effectiveness, additive effects, comparative effects, and user characteristics.
  4. Use of Gray et al.’s ontology to standardize DMP classification.

Results

• Concrete findings:
  • 85% of experiments testing DMP effects found statistically significant behavioral changes, with relative increases in undesirable outcomes ranging from 112% to 1500% (mean = 211%).
  • Privacy-related harms (e.g., increased data sharing) were the most commonly measured outcomes.
  • Interventions (e.g., educational tools, reflective interstitials) were largely ineffective, with only 4 out of 27 experiments showing partial success.
  • Evidence on additive effects of multiple DMPs and comparative effectiveness of different DMP types is inconclusive.
  • Personal characteristics (e.g., age, gender, privacy concerns) showed limited and inconsistent correlations with DMP effects.
• Advantage over baselines: The review consolidates fragmented experimental evidence, providing a clearer picture of DMP effects and highlighting the ineffectiveness of current interventions.
• Experiments / evaluation:
  • Domains: consent popups, subscriptions, privacy settings, e-commerce, streaming, advertising, donations.
  • Recruitment methods: online platforms, university mailing lists, live A/B tests.
  • Sample sizes ranged from 40 to 2.6 million participants.
  • Metrics: statistical significance, effect sizes, and behavioral outcomes.
• Limitations and future work:
  • Limited research on long-term and societal harms of DMPs.
  • Sparse evidence on additive effects and comparative effectiveness of DMP types.
  • Underrepresentation of certain domains (e.g., attention and autonomy harms).
  • Need for experiments capturing extended timescales and repeated interactions.

Summary

This systematic review synthesizes experimental evidence on the behavioral effects of dark patterns (DMPs) and the limited success of interventions designed to mitigate them. It finds strong agreement that DMPs significantly alter user behavior, often leading to privacy invasions or unintended purchases, with interventions proving largely ineffective. The review highlights gaps in understanding additive effects, comparative effectiveness, and long-term harms, urging further research and regulatory action to address these manipulative practices. By consolidating evidence, the study provides a foundation for policymakers and researchers to develop more effective strategies against DMPs.