Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit

Title of the Paper

Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit

Paper Information

• Domain: Human-Computer Interaction, AI Design, and Healthcare Innovation
• Keywords: Brainstorming, Concept Generation, Human-Computer Interaction Design, Healthcare, AI Failure Rate, High-Risk Applications
• Conference: CHI Conference on Human Factors in Computing Systems (CHI ’24)

Research Background and Problem Statement

• Problems and Challenges:

  • Although AI technologies perform well in laboratory settings, they often fail in real-world applications, especially in healthcare.
  • The practicality and usability of clinical AI systems are often unclear, leading to reluctance among clinicians to adopt these technologies.
  • Multidisciplinary teams frequently face challenges during the early stages of AI project development, including problem definition and concept generation.

• Significance:

  • Healthcare is a high-risk domain where AI failures can have severe consequences (e.g., delayed treatment, incorrect predictions).
  • Currently, 85% of AI projects fail during the pre-deployment phase, resulting in significant waste of resources and time, necessitating fundamental solutions to address this issue.

• Motivation and Related Work:

  • Based on findings from HCI research, a human-centered and participatory approach is proposed to improve AI development.
  • Emerging design practices increasingly integrate technological capabilities with user needs to generate solutions requiring lower AI performance while delivering significant value.
  • Systematic exploration of the healthcare domain (e.g., ICU) is conducted to complement existing research on the causes of AI failures.

Solution

• Methodology and Innovation:

  • Three-Stage Design Process:
    • Stage 1: Brainstorming:
      • Initial design activities are user-centered, followed by a "matching approach" based on AI capabilities to expand the design space.
      • Abstracting AI capabilities and providing examples help teams understand the possibilities and limitations of the technology.
      • Evaluation of ideas using Impact-Investment Matrix and Task Expertise-Model Performance Matrix ensures the outcomes include low-risk, feasible solutions.
    • Stage 2: Problem Definition:
      • The Do-Reason-Know worksheet clarifies the interaction forms, model reasoning, and data requirements for each concept.
      • In-depth exploration of 12 concepts (e.g., pre-ordering medications, patient prioritization) examines feasibility and potential technical implementation challenges.
    • Stage 3: Concept Design and Co-Design:
      • Selection of an AI concept predicting patient suitability for SAT/SBT (spontaneous breathing trials) and creation of a low-fidelity prototype.
      • Collaboration with nurses and respiratory therapists to collect feedback and refine the design further.

• Key Technologies and Practices:

  • Development of AI capability abstractions and case resources to support heuristic modeling and concept generation for multidisciplinary teams.
  • Use of phased worksheets (Do-Reason-Know) to specify detailed requirements for AI concepts, particularly in data quality and risk assessment of model behavior.
  • Iterative questioning to simplify concepts and reduce model performance requirements, breaking complex problems into low-risk steps.

Research Outcomes

• Specific Results:

  • Multidisciplinary Team Collaboration Framework: Successfully designed a system that facilitates collaboration during early AI innovation stages, including brainstorming, problem definition, and concept design.
  • Tools and Methods: Created AI capability and example resource sets, Do-Reason-Know design templates, and evaluation matrices, significantly improving the quality of concept generation and selection by teams.
  • Example Concept Designs:
    • Predicting patient eligibility for SAT/SBT trials to simplify clinical team decision-making.
    • Predicting medication delivery times to reduce pharmacy delays and improve patient care efficiency.

• Advantages:

  • Avoids the pitfalls of traditional user-centered design methods (e.g., generating concepts that are either unfeasible or do not require AI).
  • Encourages effective exploration of technological possibilities by multidisciplinary teams, expanding the design space and reducing the occurrence of high-risk concepts.

• Experimental and Evaluation Results:

  • The second brainstorming session was more effective than the first, generating concepts with broader design space coverage and lower implementation difficulty.
  • Most new designs utilized existing ICU datasets to generate clinically valuable solutions.
  • The final SAT/SBT trial prototype was preliminarily endorsed by nursing staff and respiratory therapists, whose feedback contributed to further design adjustments.

• Limitations and Future Directions:

  • Limitations: Lack of follow-up evaluation for all generated concepts; unverified practical effectiveness of design ideas; potential sequence effects in the design methodology.
  • Future Directions:
    • Explore how to expand multidisciplinary design resources to support AI innovation in other high-risk domains.
    • Develop new evaluation matrices to address additional AI dimensions, such as risk, model fairness, and privacy concerns.
    • Advance from concept generation to prototype development and parallel testing, assessing systematic improvements to the design process.