Birds of a Feather Don't Fact-check Each Other: Partisanship and the Evaluation of News in Twitter's Birdwatch Crowdsourced Fact-checking Program

Title of the Paper

Birds of a Feather Don’t Fact-Check Each Other: Partisanship and the Evaluation of News in Twitter’s Birdwatch Crowdsourced Fact-Checking Program

Paper Information

• Subject Area: Social Media, Crowdsourced Fact-Checking, Partisan Bias
• Keywords: Social Media, Crowdsourcing, Fact-Checking, Partisan Bias, Trust Evaluation, Misinformation, Content Assessment, Birdwatch Platform, Helpfulness Ratings, Information Filtering

Research Background and Problem

• Problem and Challenges: How partisan bias on social media influences users' judgments of information accuracy and their evaluation of the helpfulness of fact-checking comments. These biases may lead to harsher evaluations of cross-partisan content and affect collaborative performance on the platform.
• Significance: Understanding the impact of partisanship on information evaluation is crucial for designing more effective social media platforms. Fact-checking outcomes can influence the accuracy of information dissemination and, consequently, public trust in information.
• Motivation and Related Work:
  • The impact of partisan conflict and the "echo chamber" phenomenon on the internet has been widely studied, but their direct effects on content evaluation remain underexplored.
  • Previous research shows that people are more likely to share news aligning with their partisan preferences rather than critically evaluating its accuracy.
  • Data from the Birdwatch platform offers a unique opportunity to directly measure evaluations of content and comments across partisan lines, filling the gap in studies that rely on behavioral data (e.g., sharing behavior) to infer user judgments.

Solution

• Proposed Approach: Using data from Twitter's Birdwatch platform to analyze whether users are more likely to label cross-partisan content as misleading and rate cross-partisan comments as unhelpful.
• Innovations:
  1. Leveraging explicitly quantified fact-checking data (evaluating "whether a tweet is misleading" and "whether a comment is helpful").
  2. Introducing a partisan scoring model (inferring users' partisan leanings based on the accounts they follow).
  3. Comparing the impact of content-related features and partisan-related features on prediction models.
• Implementation Techniques and Steps:
  • Data Sources: Includes tweet text data, Birdwatch comment data, and helpfulness rating data.
  • Models: Random forest and logistic regression models to predict user evaluation behavior.
  • Feature Extraction: Content features such as tweet and comment length, sentiment, readability, and number of URLs; contextual features such as users' partisan scores, follower count, and posting frequency.
  • Analysis: Assess whether partisan preferences influence the classification of tweets as misleading and the helpfulness ratings of comments.

Research Findings

• Key Results:

  • Users are more likely to negatively evaluate cross-partisan content. For example, the probability of labeling cross-partisan tweets as "misleading" is significantly higher than for same-partisan content.
  • Users are more inclined to rate cross-partisan fact-checking comments as "unhelpful" compared to same-partisan comments.
  • Models indicate that partisan-related features are more predictive of user behavior than tweet or comment text content features.

• Advantages Over Existing Solutions:

  • By emphasizing contextual features (e.g., partisan scores), the study captures the biases underlying user evaluations.
  • The analysis of triadic partisan interactions (among the tweet author, commenter, and rater) provides a more comprehensive understanding of partisan dynamics.

• Experimental and Evaluation Results:

  • The random forest model achieved the highest predictive performance (AUC > 0.84) when incorporating partisan features.
  • Partisan interaction effects on helpfulness ratings were significant, with a positive correlation between same-partisan ratings and positive evaluations.
  • There is a strong linear relationship between helpfulness ratings and the proportion of same-partisan comments (R²=0.42).

• Limitations and Future Directions:

  • Birdwatch platform users are notably unrepresentative (e.g., a higher proportion of male users and more active politically extreme users).
  • The study does not directly verify the subjective or objective accuracy of "misleading content" or "unhelpful comments."
  • Future directions include:
    1. Examining partisan interaction dynamics in a broader social media ecosystem.
    2. Evaluating the participation effects of a more diverse user base.
    3. Exploring improvements to the helpfulness rating system to mitigate the impact of partisan bias on collaboration and content evaluation.