Governance of AI-Generated Content: A Case Study on Social Media Platforms

Paper Title

Governance of AI-Generated Content: A Case Study on Social Media Platforms

Publication Info

• Topic area: Governance frameworks for AI-generated content on social media platforms.
• Keywords: AI-generated content, social media governance, content moderation, generative AI, platform policies, AI disclosure, user-generated content, deepfakes, AI tools, content labeling.

Background and Problem

• Problem / challenge: The rapid increase in AI-generated content on social media platforms raises concerns about trustworthiness, authorship, copyright, and community safety. Existing governance mechanisms are often insufficient or inconsistent, and there is limited understanding of platform-driven governance responses.
• Significance: Effective governance is critical to ensure safe and trustworthy online environments, protect user rights, and address emerging risks such as misinformation, deepfakes, and low-quality AI-generated content.
• Motivation and related work: Previous studies have explored AI-generated content's impact on online communities and bottom-up governance approaches (e.g., Reddit), but there is limited research on platform-driven, top-down governance. This study aims to fill this gap by systematically analyzing governance practices across 40 major social media platforms.

Solution

• Proposed approach: Systematic analysis of AI-generated content governance across 40 popular social media platforms, focusing on policies, enforcement strategies, and user resources.
• Novelty:
  1. Creation of an annotated dataset of platform responses to AI-generated content.
  2. Identification of six governance approaches for AI-generated content.
  3. Recommendations for more explicit, comprehensive, and forward-looking governance frameworks.
• Procedure and key techniques:
  • Selection of 40 representative social media platforms based on popularity and diversity.
  • Web scraping and manual curation of 2,518 pages, filtered to 361 pages from 29 platforms explicitly addressing AI-generated content.
  • Thematic analysis of governance actions using a structured codebook derived from prior content moderation frameworks.

Results

• Concrete findings:
  • 27/40 platforms explicitly govern AI-generated content, while 13 rely on existing policies without specific mention of AI.
  • Six governance approaches identified:
    1. Moderating AI-generated content that violates existing policies (25 platforms).
    2. Disclosing and labeling AI-generated content (18 platforms).
    3. Restricting posting and sharing of AI-generated content (5 platforms).
    4. Constraining monetization of AI-generated content (6 platforms).
    5. Controlling output generation and distribution from integrated AI tools (14 platforms).
    6. Educating and empowering users about AI-generated content (17 platforms).
  • Platforms with integrated AI tools often prioritize safety measures for tool outputs and their distribution.
• Advantage over baselines:
  • Provides the first cross-platform analysis of AI-generated content governance, identifying gaps and inconsistencies in current practices.
  • Highlights the need for tailored governance strategies addressing quality, ownership, and user empowerment.
• Experiments / evaluation:
  • Dataset analysis of 361 pages from 29 platforms.
  • Thematic coding of governance actions, focusing on rule definitions, detection methods, and enforcement.
• Limitations and future work:
  • Data represents a one-time snapshot and excludes smaller platforms or non-public documents.
  • Future work should include longitudinal studies, in-depth platform-specific analyses, and evaluations of governance mechanisms.

Summary

This study systematically examines how 40 popular social media platforms govern AI-generated content, identifying six governance approaches ranging from moderating inappropriate content to empowering users with tools and education. While two-thirds of platforms explicitly address AI-generated content, policies are often inconsistent and lack comprehensive frameworks. Platforms with integrated AI tools tend to emphasize safety and control over tool outputs. The findings highlight significant gaps in disclosure standards, detection reliability, and governance of quality and ownership. Recommendations include adopting explicit policies, improving detection and labeling systems, and providing user education and controls. These insights inform future governance strategies for AI-generated content in online spaces.