Underreporting of AI Use: The Role of Social Desirability Bias

Paper Title

Underreporting of AI Use: The Role of Social Desirability Bias

Publication Info

• Topic area: Measurement and social dynamics of AI adoption in educational settings.
• Keywords: AI adoption, social desirability bias, indirect questioning, self-reporting bias, generative AI, education policy, survey methodology, stigma, peer perception, organizational norms.

Background and Problem

• Problem / challenge: Self-reported surveys on AI adoption yield inconsistent results, with discrepancies of up to 40 percentage points. Social desirability bias (SDB) may distort these measures, leading to underreporting or overreporting of AI use.
• Significance: Accurate measurement of AI adoption is critical for developing effective policies, fostering responsible use, and addressing social norms around AI in educational and organizational settings.
• Motivation and related work: Prior studies have documented SDB in sensitive topics like compliance and ethics, but existing methods for measuring AI use fail to account for SDB. Indirect questioning, a method from psychology, offers a potential corrective by eliciting peer-reported behaviors.

Solution

• Proposed approach: Indirect questioning methodology to measure AI adoption while mitigating social desirability bias.
• Novelty:
  1. Demonstrates the use of indirect questioning to uncover discrepancies in self- and peer-reported AI use.
  2. Provides qualitative evidence linking reporting gaps to stigma and social norms around AI.
  3. Offers recommendations for adapting AI policies based on revealed social dynamics.
• Procedure and key techniques:
  • Conducted two surveys: one representative sample of 338 university students and a follow-up sample of 96 students.
  • Participants reported both their own AI use (direct questioning) and their peers’ AI use (indirect questioning).
  • Analyzed reporting gaps using logistic regression and qualitative coding of free-text responses.
  • Explored alternative explanations for reporting gaps, such as availability bias and information asymmetry.

Results

• Concrete findings:
  • Self-reported AI use was 60%, while peer-reported use was 90%, revealing a 40% gap.
  • Logistic regression confirmed the reporting gap was statistically significant (p < 0.001).
  • 79% of participants attributed the gap to underreporting their own use, with 70% citing embarrassment or stigma as the primary reason.
• Advantage over baselines: Indirect questioning revealed biases that traditional self-report methods could not, providing a more nuanced understanding of AI adoption rates and social norms.
• Experiments / evaluation:
  • Survey 1: Representative sample from the University of Chicago, using direct and indirect questioning.
  • Survey 2: Follow-up with Prolific participants to explore mechanisms behind the reporting gap.
  • Metrics: AI reliance, frequency of use, and qualitative explanations for reporting discrepancies.
• Limitations and future work:
  • Sample skewed toward affluent students at a private university, limiting generalizability.
  • Social desirability bias may manifest differently in non-educational settings, such as firms.
  • Future studies should replicate the methodology across diverse institutions and contexts.

Summary

This study investigates the role of social desirability bias in distorting self-reported measures of AI adoption. Using indirect questioning, researchers found a significant gap between self-reported (60%) and peer-reported (90%) AI use among university students, primarily driven by stigma and concerns about academic integrity. Qualitative analysis highlighted embarrassment, shame, and availability bias as key factors behind the reporting gap. The findings suggest that current statistics on AI adoption may be unreliable, with implications for policy design in educational and organizational settings. Indirect questioning provides a corrective methodology, enabling institutions to better understand social norms and strategically plan AI integration and training programs.