Personal Validation Effect in LLMs: Positive AI Responses Bias Perceptions of Validity, Reliability, Personalization, and Usefulness of Fictitious Predictions

Paper Title

Personal Validation Effect in LLMs: Positive AI Responses Bias Perceptions of Validity, Reliability, Personalization, and Usefulness of Fictitious Predictions

Publication Info

• Topic area: Psychological biases in human-AI interaction, focusing on the personal validation effect in LLM-generated predictions.
• Keywords: Personal validation effect, positivity bias, human-AI interaction, cognitive biases, LLMs, AI-generated predictions, personalization, reliability, user perception, decision-making.

Background and Problem

• Problem / challenge: While the personal validation effect has been extensively studied in non-AI contexts, its role in shaping user perceptions of AI-generated predictions remains underexplored. Existing research has not systematically examined how positive or negative framing of AI predictions influences perceived validity, reliability, personalization, and usefulness.
• Significance: Understanding this phenomenon is critical as AI systems increasingly influence decision-making in high-stakes domains such as healthcare, education, and finance. Misjudgments caused by positivity bias could lead to over-reliance on AI, poor decisions, and ethical concerns.
• Motivation and related work: Prior studies have shown that people are susceptible to positivity bias and personal validation in contexts like astrology and personality assessments. However, research on how these biases manifest in AI interactions, particularly with LLM-generated predictions, is limited. This paper bridges the gap by empirically testing the effect of prediction valence on user perceptions and exploring moderating factors.

Solution

• Proposed approach: The study investigates the personal validation effect in AI-generated predictions using a simulated investment game. Participants were exposed to fictitious positive or negative predictions from AI, astrology, and personality-based sources and rated these predictions on perceived validity, personalization, reliability, and usefulness.
• Novelty:
  1. Demonstrates that the personal validation effect applies robustly to LLM-based predictions, with positive predictions perceived as significantly more favorable.
  2. Explores individual differences (e.g., cognitive style, paranormal beliefs, gullibility, AI attitudes, age) as moderators of the effect.
  3. Provides design guidelines to mitigate cognitive biases in human-AI interaction.
  4. Extends prior research on placebo effects and sycophantic behavior in AI systems by showing how prediction valence alone influences user judgments.
• Procedure and key techniques:
  • Participants (N=238) played a 10-round simulated investment game and received fictitious predictions (positive or negative) from three sources (AI, astrology, personality).
  • Predictions were rated on validity, personalization, reliability, and usefulness using 7-point Likert scales.
  • Mixed-effects models analyzed the effects of prediction valence and moderating factors (e.g., cognitive style, gullibility, AI attitudes).

Results

• Concrete findings:
  • Positive AI-generated predictions were rated as 36% more valid, 42% more personalized, 27% more reliable, and 22% more useful than negative predictions.
  • The positivity bias was consistent across all prediction sources (AI, astrology, personality).
• Advantage over baselines:
  • AI predictions were rated higher in perceived validity compared to astrology-based predictions but were similar to personality-based predictions.
  • Positive framing amplified perceived personalization more than other subscales.
• Experiments / evaluation:
  • Participants were randomly assigned to positive or negative prediction groups, with predictions presented in a single-turn, non-interactive format.
  • Moderating factors like cognitive style, paranormal beliefs, gullibility, AI attitudes, and age influenced sensitivity to prediction valence.
• Limitations and future work:
  • Single-turn interactions may not capture longitudinal effects of positivity bias.
  • Uniform prediction format limits generalizability to diverse AI interfaces (e.g., avatars, speech-based systems).
  • Future research should explore behavioral outcomes, cross-cultural differences, and mitigation strategies for biases.

Summary

This study demonstrates that the personal validation effect and positivity bias significantly influence user perceptions of AI-generated predictions, with positive predictions rated as more valid, personalized, reliable, and useful than negative ones. Individual differences such as cognitive style, paranormal beliefs, gullibility, and AI attitudes moderate these effects. The findings highlight the need for ethical AI design to mitigate cognitive biases and ensure balanced human-AI collaboration in high-stakes applications. Future work should explore longitudinal interactions, diverse AI interfaces, and behavioral impacts to refine mitigation strategies.