Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens

Document Title

Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens

Document Information

• Topic Area: Application of artificial intelligence in medical decision-making, particularly trust and collaboration in time-constrained environments
• Keywords: medical decision support tools, major depressive disorder, collaborative design, explainable AI, clinical workflow integration, patient preferences, time constraints, trustworthy systems, domain knowledge, randomized clinical trials

Research Background and Problem

• Problem or Challenge:

  • Major depressive disorder (MDD) is a complex condition with challenging treatment options and a lack of clear clinical guidance.
  • Existing machine learning-based decision support tools (DSTs) often fail in clinical practice due to poor system integration and low user acceptance.
  • Selecting antidepressant medications typically relies on trial-and-error methods, leaving many patients without symptom relief.

• Significance:

  • MDD has a high prevalence rate, imposing a significant burden on patients and society.
  • Improved tools can assist healthcare providers in making more effective treatment decisions, enhancing patient outcomes and reducing the time and cost of trial-and-error approaches.

• Research Motivation and Related Work:

  • Current DST designs often overlook user needs, focusing on model accuracy rather than practical application scenarios.
  • In complex medical environments, machine learning tools must integrate with sociotechnical systems, clinical routines, patient preferences, and resource constraints.
  • Related studies highlight the critical importance of a sociotechnical perspective for effective integration of medical tools.

Solution

• Proposed Solution:

  • The authors employed an iterative collaborative design process, working with clinicians to develop a prototype DST for antidepressant treatment decisions.
  • The tool is designed as a multi-user system, supporting collaborative decision-making between patients and providers, with on-demand explanation features tailored to the time-constrained clinical environment.

• Innovations:

  • Introduced a sociotechnical perspective, emphasizing social, technical, and organizational factors in medical processes.
  • Proposed a differentiated on-demand explanation mechanism compared to existing clinical guidelines, avoiding overly complex explanations for each prediction.
  • The tool design not only focuses on individual users (clinicians) but also addresses patient participation needs, team collaboration, and integration with existing resources.

• Implementation Steps and Key Technologies:

  • Conducted semi-structured interviews and focus groups to explore user needs and design the prototype tool.
  • Core functionalities include: machine learning-based medication recommendations, treatment predictions (e.g., stability scores, dropout probability scores), and patient interaction interfaces.
  • Collected feedback from clinicians to redesign the initial prototype for better support of collaborative decision-making and adaptation to real-world environments.

Research Outcomes

• Outcomes:

  • User research identified clinicians' expectations and needs for DSTs, including patient collaboration support, integration with existing workflows, and tool validation.
  • Proposed specific design principles to improve DSTs, such as interactive patient record adjustments, clear medication comparison displays, and provision of on-demand validation information.

• Advantages:

  • Compared to traditional tools focused on model performance, this design better meets the needs of real clinical environments.
  • Supports actionable recommendations, improving treatment outcomes and patient engagement.

• Experiments and Evaluation Results:

  • Clinicians found the prototype effective in simplifying treatment decisions and providing useful recommendations.
  • Sample demonstrations revealed that when model outputs conflicted with clinicians' knowledge or expectations, the lack of differentiated explanation mechanisms reduced confidence in using the tool.

• Limitations and Future Directions:

  • Limitations include the tool design primarily focusing on clinicians' perspectives, with insufficient consideration of patient and other healthcare professionals' involvement.
  • Future work will expand to patient collaborative design and multi-user system development.
  • Further exploration is needed for designs adaptable to different contexts, such as high-risk decision-making or resource-constrained settings.

Through this study, the authors propose new directions for designing and deploying AI tools in medical decision-making, emphasizing the importance of sociotechnical factors and collaboration among patients and medical teams. These insights offer valuable implications for tool design in other medical domains.