Large Language Model (LLM)-driven Adversarial Social Influences in Online Information Spread: Risks and Interventions

Paper Title

Large Language Model (LLM)-driven Adversarial Social Influences in Online Information Spread: Risks and Interventions

Publication Info

• Topic area: The impact of LLM-driven adversarial social bots on misinformation spread and potential interventions.
• Keywords: LLM, adversarial social influence, misinformation, social bots, credibility prompts, online information spread, cognitive interventions, political bias, AI-generated content, misinformation detection.

Background and Problem

• Problem / challenge: The rise of LLM-driven social bots enables scalable and realistic manipulation of online discourse, distorting truth and influencing user judgments about misinformation. Existing interventions often overlook the role of manipulated social influences in misinformation spread.
• Significance: Adversarial social influences can impair users' ability to discern true from false information, increase misinformation spread, and undermine trust in online platforms, posing risks to public discourse and democratic processes.
• Motivation and related work: Prior work has shown the role of social bots in amplifying misinformation and the effectiveness of interventions like fact-checking and credibility prompts. However, these approaches often focus on content-level interventions and fail to address adversarial social influences created by LLM-driven bots. This paper addresses this gap by studying the effects of such influences and testing scalable interventions.

Solution

• Proposed approach: The study investigates the effects of LLM-driven adversarial social influences on misinformation detection and sharing and evaluates two lightweight interventions: AI-generated content detectors and warning prompts.
• Novelty:
  1. Empirical demonstration of how LLM-driven adversarial social influences impair users' ability to detect misinformation and discern true from false information.
  2. Evaluation of credibility prompts as scalable interventions to mitigate the effects of adversarial social influences.
  3. Analysis of contextual factors (e.g., political alignment, format of influence) that moderate the effectiveness of interventions.
  4. Exploration of how adversarial social influences affect users' expressions and the broader information environment.
• Procedure and key techniques:
  1. Conducted two pre-registered, randomized experiments with 176 and 155 participants, respectively, using a dataset of 30 political news stories (17 true, 13 false).
  2. Experiment 1 examined the effects of adversarial comments and replies on misinformation detection and sharing.
  3. Experiment 2 tested the effectiveness of two credibility prompts (detector and warning) in mitigating these effects.
  4. Analyzed participants' veracity judgments, sharing intentions, and textual responses to assess the impact of interventions and adversarial influences.

Results

• Concrete findings:
  • Adversarial social influences significantly reduced participants' accuracy in detecting misinformation (e.g., Accuracy: F(2, 1933) = 12.28, p < 0.001) and their discernment in sharing true versus false content.
  • Credibility prompts improved misinformation detection accuracy (e.g., F(2, 1922) = 5.10, p = 0.006) but had limited effects on sharing intentions.
  • Adversarial comments had a stronger negative impact on sharing real news than adversarial replies.
• Advantage over baselines:
  • Both credibility prompts (detector and warning) significantly improved misinformation detection compared to no intervention (e.g., detector: p = 0.027, warning: p = 0.013).
  • Adversarial influences were more harmful when participants' political leanings conflicted with the news content.
• Experiments / evaluation:
  • Experiment 1: Tested the effects of adversarial comments and replies on veracity judgments and sharing intentions.
  • Experiment 2: Evaluated the effectiveness of credibility prompts in mitigating these effects under adversarial social influences.
  • Metrics: Accuracy, truth discernment, intention to share, sharing discernment.
• Limitations and future work:
  • Limited generalizability to real-world social media environments due to controlled experimental settings.
  • Focused on binary veracity judgments of concise political news; future work should examine mixed-veracity content and diverse topics.
  • Did not explore more complex bot behaviors or long-term effects of adversarial influences.
  • Future research should investigate adaptive and context-aware interventions, longitudinal effects, and theoretical mechanisms of adversarial social influence.

Summary

This paper investigates the risks posed by LLM-driven adversarial social influences on misinformation spread and evaluates two lightweight interventions—AI-generated content detectors and warning prompts. The experiments demonstrate that adversarial social influences impair users' ability to detect misinformation and discern true from false information, with stronger effects when users' political leanings conflict with the news. Credibility prompts improve misinformation detection but have limited impact on sharing intentions. The findings highlight the need for scalable, context-aware interventions to combat adversarial social influences and suggest directions for future research on mitigating misinformation in the era of LLMs.