Beyond the Checkbox: Strengthening DSA Compliance Through Social Media Algorithmic Auditing

Paper Title

Beyond the Checkbox: Strengthening DSA Compliance Through Social Media Algorithmic Auditing

Publication Info

• Topic area: Algorithmic auditing for compliance with the Digital Services Act (DSA).
• Keywords: DSA, algorithmic auditing, AI systems, social media platforms, recommender systems, compliance, transparency, user protection, minors, targeted advertising.

Background and Problem

• Problem / challenge: Current auditing practices under the DSA are insufficient for assessing dynamic AI systems, leading to inconsistencies in methodologies and outcomes.
• Significance: Ensuring compliance with DSA obligations is critical for safeguarding user rights, transparency, and mitigating systemic risks posed by Very Large Online Platforms (VLOPs) and Very Large Online Search Engines (VLOSEs).
• Motivation and related work: Traditional audits, rooted in financial and IT systems, fail to address the complexities of AI-driven platforms. Prior research highlights methodological gaps, including reliance on platform-provided data and limited socio-technical perspectives, necessitating new approaches.

Solution

• Proposed approach: Algorithmic auditing – a behavioural assessment method that simulates user interactions to evaluate AI system responses and compliance.
• Novelty:
  1. Systematic analysis of DSA audit reports to identify methodological inconsistencies and limitations.
  2. Proposal of algorithmic auditing as a complementary methodology to traditional audits.
  3. Integration of human-computer interaction (HCI) principles into auditing practices for socio-technical transparency.
• Procedure and key techniques:
  • Define audit scenarios based on user profiles and actions.
  • Simulate user behaviour using bots or human agents to interact with platforms.
  • Record and analyze platform responses for compliance with DSA provisions.
  • Produce structured audit reports highlighting findings and recommendations.

Results

• Concrete findings:
  • Audits reveal significant methodological variations across platforms, including differences in assessing recommender system transparency, protection of minors, and advertising restrictions.
  • TikTok auditors acknowledged limitations in assessing dynamic system behaviour, while other platforms received positive compliance conclusions despite similar challenges.
• Advantage over baselines: Algorithmic auditing addresses structural limitations of traditional audits, such as temporal constraints and lack of socio-technical framing, enabling long-term and scalable assessments.
• Experiments / evaluation:
  • Analysis of four audit reports from YouTube, Facebook, Instagram, and TikTok covering three DSA provisions (Articles 27, 28, and 26).
  • Comparative document analysis using thematic coding to identify methodological inconsistencies and challenges.
• Limitations and future work:
  • Algorithmic audits require significant technical expertise and may oversimplify complex real-world environments.
  • Challenges include designing authentic audit scenarios and improving replicability.
  • Future research should focus on multidisciplinary approaches to refine audit methodologies and address identified gaps.

Summary

This paper systematically analyzes the first wave of DSA audit reports, highlighting significant methodological inconsistencies and limitations in assessing AI systems. It proposes algorithmic auditing as a complementary approach to traditional audits, leveraging behavioural assessments to address dynamic and socio-technical challenges. Algorithmic auditing enables long-term, scalable evaluations of compliance with DSA provisions, particularly for recommender systems, minors’ protection, and advertising restrictions. While promising, its adoption requires overcoming technical and methodological challenges, offering potential for multidisciplinary research to enhance regulatory enforcement and user protection.