Understanding the Impact of Explanations on Advice-Taking: a User Study for AI-based Clinical Decision Support Systems

Title of the Paper

Understanding the impact of explanations on advice-taking: A user study for AI-based clinical Decision Support Systems

Paper Information

• Field of Study: Application of Explainable AI (XAI) in Clinical Decision Support Systems
• Keywords: XAI, eXplainable AI, HCI, user study, behavioral intention, trust, advice-taking, clinical decision support systems

Research Background and Issues

• Identified Problems or Challenges:

  1. AI has significant potential as an assistive tool in the medical field, but the primary challenge lies in the black-box nature of models, which hinders trust and adoption by physicians.
  2. Many existing XAI methods lack testing in real-world user scenarios and are not designed with user needs in mind.
  3. AI explanations in the medical domain need to foster trust among physicians and enable them to properly calibrate their trust in system recommendations. However, current research often overlooks these practical requirements.

• Significance of the Research:

  • Achieving explainability in AI systems can help physicians build accurate mental models, increase trust in AI recommendations, and thereby improve system adoption and acceptance.
  • In high-risk domains like healthcare, the reliability and explainability of AI are critical, not only for technology adoption but also for ethical and legal considerations.

• Motivation and Related Work:

  • XAI is crucial for enhancing the transparency and interpretability of AI recommendations. Existing XAI methods (e.g., LIME and SHAP) provide general model explanations but are rarely tailored for specific application domains like medicine.
  • Physicians often avoid using algorithms due to a lack of confidence or may over-rely on AI results due to overtrust (e.g., "automation bias").
  • This study aims to explore how explainability affects advice-taking and trust in medical decision-making through a user study.

Solution

• Research Methods:

  • An online user study was conducted to compare two AI interfaces: one providing only recommendations and the other providing both recommendations and explanations.
  • A customized XAI method, "Doctor XAI," was employed to ensure that the explanation process and outputs were clinically meaningful.
  • Within an estimation task framework, medical professionals were asked to predict myocardial infarction probabilities based on patient data and AI recommendations, observing changes in their decisions and trust in the system.

• Key Innovations:

  • The study incorporated the "Judge-Advisor System" framework to design experimental scenarios, collecting comparative data on how AI interfaces with and without explanations influence behavioral intentions and trust levels.
  • Quantitative metrics such as Weight of Advice (WOA) and explanation satisfaction scales were introduced, alongside open-ended feedback to gather qualitative insights into user experience.
  • Focused on users in the medical data domain, the study closely aligned XAI methods with the needs of actual healthcare practitioners.

• Experimental Design:

  • Participants included 31 medical professionals, such as doctors, nurses, and nursing assistants. The online experiment was conducted via the Prolific platform.
  • The study employed a comparative design of two interfaces, ensuring that learning effects and order effects did not bias the data.
  • Data collected included behavioral intention, WOA, trust levels, perceived explanation quality, and open-ended feedback.

Research Findings

• Specific Results:

  • Comparative analysis revealed:
    • The AI interface with explanations (Dr. XAI) had a greater impact on user decisions compared to the interface providing only recommendations (Dr. AI), with a significant increase in WOA.
    • Explanation quality was positively correlated with behavioral intention and explicit trust, but this specific XAI explanation did not significantly enhance behavioral intention or trust levels.
    • Explanations were ultimately deemed unsatisfactory, while simple recommendations were considered overly generic.
  • Interfaces with explanations may help prevent errors in collaborative decision-making between novices (e.g., doctors and nurses).

• Advantages:

  • Provided extensive user feedback, contributing to the improvement of existing XAI designs.
  • Combined quantitative and qualitative data, offering valuable references for evaluating and optimizing XAI.

• Limitations and Future Directions:

  1. Sample Size: The study involved only 31 participants. Future research requires larger sample sizes to enhance the reliability of conclusions.
  2. Single Explanation Method: This study tested only feature-removal-based explanations. Future work plans to explore other types of explanations.
  3. Uncovered Error Scenarios: The study tested only correct algorithm recommendations, leaving trust and behavioral responses to incorrect predictions unexplored.
  4. AI Replacement Concerns: A key concern in user feedback was the fear of AI replacing humans. This has profound implications for the acceptance of XAI in medical contexts, warranting further sociocultural research.