Human-Algorithmic Interaction Using a Large Language Model-Augmented Artificial Intelligence Clinical Decision Support System

Title of the Paper

Human-Algorithmic Interaction Using a Large Language Model-Augmented Artificial Intelligence Clinical Decision Support System

Paper Information

• Subject Area: AI algorithms in medical workflows, focusing on clinical decision support systems
• Keywords: Health clinical, machine learning, medicine: geriatric care/hospitals, qualitative methods, quantitative methods, artificial intelligence, clinical decision support system, workflow, electronic health records

Research Background and Problems

• Identified Issues or Challenges:

  • AI-driven clinical decision support systems (AI-CDSS) have limited effectiveness in optimizing clinical outcomes.
  • Insufficient research on how healthcare providers interact with AI systems.
  • Although large language models (LLMs) have the potential to improve AI-CDSS, they have rarely been validated in simulated or real clinical scenarios.

• Significance:

  • Efficiency and accuracy in clinical decision-making are critical for patient management and healthcare resource utilization.
  • The healthcare industry's low trust in AI technology hinders the widespread adoption of new tools.

• Research Motivation and Related Work:

  • Previous studies have predominantly focused on single-user interactions, with limited exploration of how medical teams utilize AI-CDSS.
  • The text generation capabilities of large language models (LLMs) can assist medical teams in making complex decisions, such as personalized recommendations for complicated patients.
  • Upper gastrointestinal bleeding (UGIB) is an acute and high-priority condition requiring collaboration among physician teams and a clear need for risk assessment tools.

Solution

• Methods or Solutions:

  • Developed a system named GutGPT: integrating a risk-prediction-based machine learning model and an LLM-based natural language Q&A functionality.
  • Used electronic health record data for binary risk prediction (high risk or low risk) for UGIB.
  • Tested the GutGPT system in simulated clinical scenarios to study its impact on physician team decision-making, trust, and interaction patterns.

• Innovations:

  • First integration of LLM into AI-CDSS, studying its usability, trust level, and interaction patterns in a simulated environment.
  • Provided real-time patient data visualization tools (interactive dashboard) and natural language responses generated by LLM to enhance communication capabilities.

• Implementation Steps and Key Technologies:

  1. GutGPT framework construction: integration of machine learning models and LLM.
  2. Data processing: dimensionality reduction and predictive model training based on patient electronic health records.
  3. System development: the dashboard offers interpretable charts for risk prediction results (e.g., PDP, ICE, and ALE).
  4. Simulation trial design: randomized grouping of physician teams, deploying different functionalities of GutGPT.

Research Outcomes

• Specific Results:

  • LLM integration made AI-CDSS more user-friendly and increased physicians' trust in the tool.
  • Data showed that participants generated an average of 3 natural language inquiries, approximately 13 words each, reflecting the system's user interaction patterns.
  • Within clinical teams, system usage and acceptance varied based on professional experience levels and role distribution.

• Advantages Compared to Existing Solutions:

  • Conventional clinical support systems are mostly rule-driven, while GutGPT achieves more intuitive and comprehensive guidance and recommendations through LLM integration.
  • LLM-generated answers combine patient data with natural language to provide detailed and easily understandable explanations.

• Experimental or Evaluation Results:

  • SUS (System Usability Scale) surveys indicated that participants generally found the system easy to use, though confidence during initial use was limited.
  • Clinical tasks and team dynamics significantly influenced usage patterns, with inquiry frequency increasing in complex decision-making scenarios.
  • Physicians with high clinical experience tended to use the system to validate their intuition, while medical students relied on the system as their primary information source.

• Limitations and Future Directions:

  • Simulated environments do not fully reflect the complexity of real clinical scenarios, such as environmental distractions and time pressures that may affect system usage.
  • Participants were all in medical training stages, lacking feedback from long-term practicing physicians.
  • Further research is needed to optimize GutGPT's interface and functionality, such as reducing response times and enhancing dashboard interpretability.
  • Future studies will explore VR/AR technologies to create more realistic simulation environments and improve dynamic performance characteristics.

Summary and Design Principles

• Summary:

  • LLM-enhanced AI-CDSS significantly improves usability and fosters trust in clinical scenarios but requires personalized deployment tailored to user backgrounds and workflows.
  • Trust can be increased through transparent and referable response data, while system acceptance can be enhanced with familiar interface designs.

• Design Principles:

  1. Comprehensive Usability Design: Improve the integration of dashboard and chat functionalities while ensuring seamless EHR compatibility.
  2. Customized Deployment Strategies: Design differentiated modes tailored to various medical specialties and user needs.
  3. Understanding Team Dynamics and Adaptive Design: Optimize the system to adapt to medical team role distribution and leadership relationships, while promoting familiarity through training.

This work provides practical design guidelines for AI applications in healthcare and lays the foundation for future research, driving medical teams to more effectively utilize AI technology to improve patient care.