Assessing enactment of content regulation policies: A post hoc crowd-sourced audit of election misinformation on YouTube

Title of the Paper

Assessing Enactment of Content Regulation Policies: A Post Hoc Crowd-Sourced Audit of Election Misinformation on YouTube

Paper Information

• Field of Study: Algorithm auditing on online platforms, regulation of election misinformation
• Keywords: misinformation, elections, voter fraud, algorithm auditing, fairness, recommendation systems

Research Background and Problem

• Problem or Challenge: As the 2022 U.S. midterm elections approach, conspiracy theories about the 2020 presidential election remain prevalent, threatening public trust in the electoral process. Although YouTube has implemented policies to reduce the spread of election-related misinformation, the effectiveness of these policies remains unclear.
• Significance: Social media platforms, especially YouTube, play a prominent role in political activities and have become major channels for spreading misinformation. Search engines and recommendation algorithms can influence user behavior and lead to the phenomenon of information filter bubbles, making it necessary to evaluate the behavior of these algorithms.
• Motivation and Related Work: While previous studies have used "sock puppet accounts" to audit YouTube's algorithms, these methods fail to fully reflect the behavior patterns of real users. This study aims to evaluate YouTube's effectiveness in regulating election misinformation through a crowd-sourced approach involving real users.

Solution

• Methodology: This study recruited 99 users through a crowd-sourcing approach and installed a self-developed browser extension tool, TubeCapture, to collect the following data over a 9-day experiment:
  • Search results for 88 election-related queries.
  • "Recommendation paths" (five layers of auto-recommended videos) for 45 seed videos.
• Innovations:
  • Conducting experiments with real user accounts instead of simulating behavior with sock puppet accounts.
  • Analyzing how YouTube's algorithms handle election-related content in various contexts, including users with different political leanings and videos with varying stances on misinformation.
  • Moving beyond studies focused on real-time events by adopting a post hoc audit approach to evaluate the governance of past events.
• Implementation Steps:
  1. Search Query Design: Relevant keywords were selected using Google Trends and YouTube video tags, filtered, and finalized into 88 search queries.
  2. Seed Video Selection: 45 videos with high view counts and election-related content were selected and categorized based on their stance as supporting, opposing, or neutral toward misinformation.
  3. Extension Tool Design and Experimental Process: The TubeCapture tool was developed to collect data, including users' search results, recommendation paths, and homepage recommendations. Standard windows (personalized results) and incognito windows (non-personalized results) were used for comparative analysis.

Research Findings

• Specific Findings:
  • Personalization of Search Results: The degree of personalization in search results was low, and most query results included videos opposing election misinformation.
  • Misinformation in Recommendation Paths:
    • When users began watching videos supporting misinformation, they were recommended a small number of videos supporting misinformation.
    • Watching videos opposing misinformation significantly increased the likelihood of entering recommendation paths containing opposing content.
    • Neutral paths had a misinformation recommendation rate between the two extremes.
  • Personalization of Recommendation Paths: The degree of personalization in recommendation paths was significantly higher than in search results and was influenced by users' subscribed channels, with independent party users being more affected.
  • Channel and Content Diversity: The overall source distribution in search results and recommendation paths was relatively fair. The Gini coefficient of channel distribution in search results indicated balanced sources, though certain channels, such as CNN and Fox News, appeared more frequently in recommendations.
• Advantages:
  • Compared to studies relying solely on sock puppet accounts, this research better reflects the impact of real users' account histories and behaviors on algorithmic recommendations.
  • Provides a post hoc auditing framework that can be used to evaluate the long-term effectiveness of online platform policies.
• Limitations and Future Directions:
  • Data Bias: Respondents were primarily white American users, which may not fully represent all demographic groups.
  • Instantaneous Deletion of Browsing History: The audit tool's capability to delete browsing history in real-time was not thoroughly tested, and future research could explore its effectiveness across different topics.
  • Seed Video Selection: The experiment relied on top popular videos as seed videos, which may not represent the specific browsing behaviors of average users.
  • Future studies could expand to cross-platform audits (e.g., Facebook, Twitter) to investigate the impact of multi-platform usage on opinion formation.

The findings indicate that YouTube has been relatively successful in controlling misinformation in search results. However, its recommendation system still has room for improvement, requiring further optimization to more effectively eliminate information bubbles.