The End of “Trust and Safety”?: Examining the Future of Content Moderation and Upheavals in Professional Online Safety Efforts

Research Background and Issues

• Identified Problems or Challenges:

  1. Between 2021 and 2023, many Trust & Safety (T&S) teams across technology platforms faced large-scale layoffs, compounded by challenges from economic downturns and political pressures. This phenomenon has significantly impacted areas such as content moderation, cybersecurity, and the control of misinformation dissemination.
  2. Public and political perceptions of T&S teams have deteriorated, particularly in the United States, where content moderation is often criticized as "censorship" and "bias" against free speech.
  3. A lack of public understanding of the true role of T&S work, coupled with widespread resource cuts, has made these teams increasingly vulnerable.

• Importance of the Issues:

  1. T&S teams are critical for ensuring the safety of online communities, controlling platform abuses (e.g., child sexual abuse material and scam content), and combating misinformation.
  2. Reductions in content moderation efforts could exacerbate issues such as the spread of misinformation, increased online harassment, and attacks on vulnerable groups, especially during critical events like elections.
  3. Negative societal perceptions of content moderation could undermine public trust in digital platforms and further hinder support for related policies or resources.

• Research Motivation and Related Work: This study, set against the backdrop of the expansion and layoffs in the T&S field, as well as societal debates on content moderation, explores:

  1. How T&S practitioners navigate professional instability and the challenges of layoffs during this period.
  2. The future development of content moderation as a mechanism for online safety.

Solutions

• Proposed Methods or Solutions: The authors conducted 20 in-depth interviews to explore the professional experiences of T&S practitioners, their perspectives on content moderation, and strategies for ensuring online safety in the context of the 2024 U.S. presidential election.

• Innovative Aspects of the Solutions:

  1. A systematic summary of the dynamic changes in the T&S field over recent years, including industry layoffs and job transitions.
  2. An exploration of the reasons behind public and legislative misunderstandings of T&S work, identifying areas for improvement.
  3. Emphasis on the trend toward professionalization in the T&S field, highlighting the importance of developing unified standards and increasing transparency.

• Implementation Steps and Key Techniques:

  1. Data Collection: Recruiting interview participants from T&S practitioners, ensuring diversity across roles such as policy-making, user experience, content moderation management, and engineering development.
  2. Data Analysis: Conducting in-depth interviews, employing open coding and axial coding to extract key themes.
  3. Synthesizing Results: Summarizing T&S practitioners' coping strategies and perspectives on the industry's future, and exploring policy and technological solutions.

Research Findings

• Specific Findings:

  1. Revealed the profound impact of widespread layoffs on the T&S industry and its practitioners, such as increased burnout and uncertainty about the industry's future.
  2. Found that the T&S field is evolving toward greater professionalization and diversification, including more consulting services and short-term contract work.
  3. Highlighted biases in public perceptions of content moderation, attributed to a lack of transparency and politicized narratives.

• Advantages Compared to Existing Solutions:

  1. Captures the long-term impacts of T&S layoffs on mental health and career development from the practitioners' perspective.
  2. Emphasizes the role of global regulatory frameworks (e.g., the EU's Digital Services Act) in providing a baseline operational guarantee for T&S teams.

• Experimental or Evaluation Results:

  • The study shows that while T&S work is often perceived externally as a "cost center," practitioners firmly believe their work is foundational to the long-term success of platforms and user safety.
  • Representative practitioners unanimously agree that strengthening regulatory frameworks and promoting the adoption of emerging technologies like AI can alleviate the pressures of content moderation.

• Limitations and Future Directions:

  1. Limitations:
     • The sample primarily comprises practitioners based in or serving the U.S. market, which may not fully reflect the diversity of the global T&S field.
     • The qualitative nature of interview data may introduce bias; future research should include larger-scale quantitative studies for validation.
  2. Future Directions:
     • Explore broader cross-cultural perspectives, particularly within decolonial frameworks in the T&S field.
     • Advance empirical research on the relationship between content moderation, platform profitability, and public safety, and strengthen the integration of T&S policies with corporate governance models.

Conclusion

This study provides a detailed account of the dynamic changes in the T&S field, including economic turbulence and societal and political divisions over content moderation. Despite facing resource cuts and external pressures, practitioners remain steadfast in their belief in the critical importance of online safety work. The authors propose strengthening the role of T&S through professionalization, increasing industry transparency, and adopting AI technologies. These findings offer valuable insights for academia, policymakers, and industry practitioners, contributing to the pursuit of safety and reliability in the ever-evolving digital information environment.