Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy

Paper Title

Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy

Publication Info

• Topic area: Human-AI collaboration and decision-making
• Keywords: AI-assisted decision-making, explainable AI (XAI), reflection, cognitive biases, over-reliance, decision accuracy, conflict detection, cognitive decoupling, human-AI interaction

Background and Problem

• Problem / challenge: Users often exhibit over-reliance or under-reliance on AI systems, even when provided with explainable AI (XAI). Existing interventions fail to fully address the cognitive biases that lead to these behaviors, particularly the lack of systematic reasoning (Type 2 processing).
• Significance: Addressing over-reliance and improving decision accuracy in AI-assisted contexts is critical for domains like medical diagnosis, hiring, and financial decision-making, where human-AI collaboration can have significant real-world impacts.
• Motivation and related work: Prior research has focused on transparency and explanation design to foster trust in AI, but these approaches often fail to mitigate biases. Theoretical models like the three-stage dual-process model highlight the importance of conflict detection and cognitive decoupling, but existing interventions do not systematically address both mechanisms.

Solution

• Proposed approach: Guided reflection—a structured intervention designed to prompt conflict detection and cognitive decoupling, encouraging systematic reasoning in AI-assisted decision-making.
• Novelty:
  1. Introduces reflection as an interaction design intervention, focusing on conflict detection and cognitive decoupling.
  2. Examines the moderating roles of individual differences, such as need for cognition and perceived understanding of AI.
  3. Provides a belief-change perspective on over-reliance and under-reliance in human-AI collaboration.
• Procedure and key techniques:
  • Participants complete a three-step reflection process:
    1. Detect conflicts between their initial judgment and AI recommendations.
    2. Decouple from intuitive responses by evaluating the accuracy of their reasoning and the AI's explanations.
    3. Justify their final decision by integrating insights from both human and AI reasoning.
  • The intervention was tested in a diabetes prediction task using SHAP explanations for XAI.

Results

• Concrete findings:
  • Reflection reduced AI over-reliance (XAI+Reflection: M=0.34) compared to XAI (M=0.58, p<0.05).
  • Reflection improved decision accuracy (XAI+Reflection: M=0.881) compared to XAI (M=0.817, p<0.05) and AI-only (M=0.810, p<0.05).
• Advantage over baselines:
  • Reflection encouraged critical evaluation of AI outputs, leading to more balanced reliance and higher decision accuracy than XAI or AI-only conditions.
  • Participants in XAI+Reflection selectively integrated AI insights, avoiding blind adherence to AI explanations.
• Experiments / evaluation:
  • A between-subjects experiment with 178 participants on a diabetes prediction task.
  • Three conditions: AI-only, XAI, and XAI+Reflection.
  • Metrics: decision accuracy, over-reliance, under-reliance, belief adjustments, and user experience measures.
• Limitations and future work:
  • Limited to SHAP explanations and a relatively simple diabetes prediction task.
  • Future work could explore other explanation types, more complex tasks, and multimodal sensing for real-time cognitive engagement.

Summary

This study introduces guided reflection as a cognitive intervention to improve decision accuracy and reduce over-reliance in AI-assisted decision-making. By prompting conflict detection and cognitive decoupling, reflection encourages systematic reasoning and belief updating. The intervention was particularly effective for individuals with a high need for cognition and a strong perceived understanding of AI. Results from a controlled experiment demonstrate that reflection significantly enhances decision accuracy and fosters more balanced reliance on AI. These findings highlight the potential of reflection to address cognitive biases in human-AI collaboration, with implications for designing more effective and personalized AI systems.