Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care

Title of the Paper

Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care

Paper Information

• Research Domain: Application of artificial intelligence in healthcare decision support, specifically studying clinicians' acceptance of AI-generated treatment recommendations and the mechanisms influencing this acceptance.
• Keywords: Human-computer interaction, healthcare, visualization, explainability, artificial intelligence, AI-assisted decision-making, clinical decision support systems, trust, collaboration, reinforcement learning

Research Background and Problem

• Challenges:
  1. Artificial intelligence (AI) has the potential to improve healthcare decision-making, but its deployment and acceptance in real-world clinical settings face significant barriers.
  2. Existing research predominantly focuses on diagnostic support, with limited attention to treatment recommendations, where "optimal decisions" often lack clear standards and are influenced by patient-specific variations.
  3. Calibrating clinicians' trust in AI is challenging, with risks stemming from over-reliance or complete disregard, directly impacting the effectiveness of AI applications.
• Significance: Improving the acceptance of AI treatment recommendations in intensive care units (ICUs) could significantly enhance patient survival rates and reduce healthcare costs.
• Motivation and Related Work:
  1. Many current AI decision support tools fail to integrate seamlessly into clinical workflows, offering limited support for actual treatment outcomes.
  2. The acceptance of AI decision tools is closely tied to clinicians' decision-making styles and subjective perceptions of the tools' trustworthiness.
  3. Enhancing AI's explainability and transparency may help reduce usage conflicts, though specific methods remain unclear.

Proposed Solution

• Approach:
  1. Developed an interactive decision support interface, "AI Clinician Explorer," to display and explain AI-generated treatment recommendations for sepsis.
  2. Utilized a reinforcement learning-based AI model to simulate patient treatment pathways using clinical data and provide recommendations to clinicians.
  3. Enhanced the interpretability of AI outputs through techniques such as visualizing relevant variables and treatment options.
• Innovations:
  1. Focused on treatment recommendations (rather than diagnostic support), an area that has been less explored.
  2. Introduced the concept of "negotiation-based" interaction strategies, emphasizing partial adoption of AI recommendations rather than binary acceptance or rejection.
  3. Combined interpretability techniques (e.g., Shapley Additive Explanations, SHAP) with dynamic patient trajectory visualizations to improve AI transparency.
• Implementation Steps:
  1. Conducted clustering analysis on historical data for 750 patient states and generated optimal choices for 25 treatment actions.
  2. Developed a comprehensive interface featuring patient browsing, trajectory visualization, recommendation explanations, and interactive tools.
  3. Used a mixed-methods study to observe the decision-making behaviors and feedback of 24 ICU clinicians under varying informational conditions.

Research Findings

• Key Results:
  1. Enhanced explainability of AI recommendations significantly increased clinicians' confidence in the tool and their perception of decision complexity.
  2. Identified four behavioral patterns in clinician-AI interactions: ignore, negotiate, consider, and rely.
     • "Ignore": Completely disregarding AI recommendations, with minimal impact on decision-making.
     • "Negotiate": Partially accepting AI recommendations while adjusting specific suggestions based on priorities.
     • "Consider": Fully adopting or rejecting AI recommendations depending on the situation.
     • "Rely": Consistently partially adopting AI recommendations.
  3. Alternative treatments and feature explanations played a crucial role in fostering partial reliance behaviors.
• Advantages Compared to Existing Solutions:
  1. Demonstrated that AI treatment tools require dynamic visualization and flexible interpretability designs more than traditional diagnostic tools.
  2. Provided a deeper exploration of the complex process and psychological impacts of clinicians accepting non-binary recommendations, compared to existing studies.
• Experimental and Evaluation Results:
  1. Explanation support in visualization conditions positively influenced clinicians' perceptions but did not significantly lead to full reliance on AI recommendations.
  2. Clinicians' trust in AI recommendations was built on data credibility, transparency, and collaborative flexibility.
• Limitations and Future Directions:
  1. Current evaluations were conducted in a single academic hospital in North America, necessitating broader sample coverage to validate the findings' generalizability.
  2. AI explanation methods and textual representations require further refinement to enhance usability under high cognitive load conditions.
  3. Future research should develop more granular behavioral and outcome metrics to capture "negotiation-based" reliance behaviors comprehensively.

Conclusion and Impact

This study provides deeper insights into the complex behavioral patterns of clinicians when accepting AI treatment recommendations in high-risk clinical environments. By aligning AI clinical support tools with clinicians' decision-making processes and psychological needs, this research could foster broader human-AI collaboration and improve patient outcomes in healthcare settings.