I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI

Title of the Paper

I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI

Paper Information

• Domain: Explainable Artificial Intelligence (XAI), Cognitive Science, Human-Computer Interaction
• Keywords: Explainable AI, Shapley values, cognitive biases, comprehension, user studies, black-box models, human-computer interaction, mental models, machine learning, transparency

Research Background and Problem

• Identified Problem or Challenge: Explainable Artificial Intelligence (XAI) systems for machine learning often provide local explanations for individual predictions to help users understand model behavior. However, these systems often neglect whether users can accurately construct a global understanding of the model. Additionally, many users overestimate their comprehension, which can lead to poor decision-making.
• Significance: As AI systems are increasingly applied in sensitive societal contexts (e.g., loan scoring, employee recruitment, crime prediction), the explainability of model behavior becomes a critical component to prevent unintended consequences, especially for non-technical users.
• Motivation and Related Work: Psychological research indicates that humans often experience the "Illusion of Explanatory Depth" (IOED) when understanding complex systems. In the context of XAI, this illusion may negatively impact users' cognitive abilities and decision-making. However, there is a lack of systematic research on whether non-technical users experience IOED when using Shapley values.

Solution

• Proposed Method or Solution: The authors designed a tabular explanation interface based on SHAP (Shapley Additive Explanations), called "SHAPTable," to help non-technical users intuitively understand machine learning model behavior.
• Innovations:
  • Developed an easy-to-use tabular explanation interface that enables users to compare and analyze multiple local explanations.
  • Employed the psychological method of "reflective self-explanation" to measure users' depth of understanding.
  • Introduced cognitive bias theories (e.g., IOED) into the user evaluation of AI system design.
• Implementation Steps and Key Techniques:
  1. Use SHAP to generate local explanations, quantifying each feature's contribution through Shapley values.
  2. Display feature contributions in a heatmap format on the interface, with tooltips for further clarification.
  3. Provide interactive features allowing users to reorder, resample, and simulate new combinations to explore the model's global behavior.
  4. Apply a mixed research methodology, including a controlled experiment with 40 participants and an online user study with 107 participants.

Research Findings

• Specific Findings:
  1. Non-technical users exhibited IOED when forming a global understanding of the model based on SHAP's local explanations, leading to overestimation of their comprehension.
  2. Users' self-assessed comprehension scores significantly decreased during the study, particularly after viewing test results.
  3. Comparisons between controlled and online experiments revealed that participants in controlled settings demonstrated deeper reflection and higher prediction accuracy.
• Advantages:
  • The study highlights the limitations of local explanations and offers recommendations for improving user interaction design in XAI systems.
  • Provides empirical insights into how to help users mitigate cognitive biases.
• Experimental or Evaluation Results:
  • Participants' comprehension scores initially increased significantly after using SHAPTable but dropped significantly post-task, supporting the IOED theory.
  • Users in controlled experiments spent more time and performed better, demonstrating a gap between subjective perception and actual understanding.
  • Reflections recorded during the study revealed that participants overly trusted simple local explanations while neglecting the impact of complex feature interactions.
• Limitations and Future Directions:
  • Limitations: The experiments were limited to a simplified scenario based on tabular data, leaving the applicability of results to other data types or domains uncertain. The sample primarily consisted of highly educated users, excluding other demographic groups.
  • Future Directions:
    1. Extend research to different AI application scenarios and user groups.
    2. Design more personalized interactive explanation tools to enable users to calibrate their understanding throughout the process.
    3. Incorporate structured processes in XAI systems to guide users in forming more accurate mental models (e.g., self-explanation training programs).