Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems

Document Title

Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems

Document Information

• Subject Area: Human-Computer Interaction, Cognitive Bias in Explainable AI, User Dependence
• Keywords: Explainable Artificial Intelligence (XAI), Cognitive Bias, Human-Computer Interaction (HCI), User Studies, Anchoring Bias, Automation Bias, Video Activity Recognition, Mental Models, Confidence Analysis

Research Background and Problem

• What problems or challenges did the authors identify?

  1. The goal of Explainable Artificial Intelligence (XAI) is to enhance user transparency and understanding of AI models, but users’ mental models may still be influenced by cognitive biases such as anchoring bias.
  2. The order in which a system’s strengths and weaknesses are presented may interfere with users’ mental models, task performance, and reliance on the system.
  3. It remains unclear whether explanation mechanisms can mitigate these cognitive biases.

• Why is this problem important?

  • Anchoring bias may lead users to either over-rely on or distrust AI systems, resulting in automation bias or potential misunderstandings, which significantly impact decision-making and task execution capabilities.
  • Developers need a clear understanding of these biases to design more optimized explainable systems.

• Research Motivation and Related Work

  • Previous studies have shown that the first impression of intelligent systems (strengths or weaknesses) can influence long-term outcomes.
  • While mechanisms related to trust and cognitive biases have been explored, few studies have delved into the sequential effects in XAI systems.

Solution

• What methods or solutions did the authors propose?

  • Utilize an explainable video activity recognition tool to study how users construct mental models under different presentation conditions.
  • Design and conduct a 2x2 controlled user experiment to investigate the effects of "system strength/weakness presentation order" and "presence or absence of explanations" on users’ mental model formation.

• What is innovative about this solution?

  • Unlike traditional studies that focus solely on model accuracy, this research explores the previously underexplored cognitive sequential effects and their interaction with explanation mechanisms.
  • The experimental design simulates real-world complexity by allowing users to freely explore scenarios while using AI decision-making tools.

• What are the implementation steps and key technologies used?

  1. Develop an explainable deep learning video activity recognition model based on the TACoS dataset, incorporating video segmentation, error detection, and local explanations to help users understand the model’s logic.
  2. Set conditions:
     • Policy order: presenting system strengths first or weaknesses first
     • Presence or absence of explanations
  3. Design open-ended tasks: verifying compliance with kitchen rules.
  4. Collect and analyze multiple metrics, including user task performance, mental model construction, and system reliance.

Research Outcomes

• What specific outcomes were achieved?

  1. Sequential Effects:
     • Users who observed system strengths first were more likely to over-rely on the system, leading to more errors.
     • Users who observed system weaknesses first exhibited lower reliance, reduced errors, but underestimated the system’s capabilities.
  2. Impact of Explanations: While explanations increased users’ confidence in tasks, they did not completely eliminate biases.
  3. Measurement Analysis: Users showed the most interest in video segments within explanations, which were more valuable than other elements.

• What advantages does it have compared to existing solutions?

  • This study not only emphasizes user task performance in XAI systems but also focuses on how users form mental models, addressing a gap in existing research on cognitive biases.

• What were the experimental or evaluation results?

  1. Task completion efficiency improved when using systems with explanations, but explanations could not fully correct biases caused by sequential effects.
  2. Regardless of whether explanations were provided, users who observed system weaknesses first had significantly lower error rates compared to those who observed strengths first.
  3. Reliance and trust were significantly influenced by cognitive anchoring, and explanation mechanisms could not completely mitigate their impact.

• Limitations and Future Directions

  • Limitations:
    1. The study focused solely on video activity recognition models, and the results may not generalize to all XAI problems.
    2. User measurements relied primarily on task performance and self-assessment, without covering more complex cognitive behaviors.
  • Future Directions:
    1. Investigate the effects of global explanations (rather than instance-level explanations) in reducing cognitive bias.
    2. Conduct broader testing and validation on different types of user interfaces or more complex AI systems.
    3. Explore strategies for correcting mental models through dynamic interaction.