"I Wouldn't Really Use It as a Practice Tool": Understanding Medical Students’ Perspectives and Needs on LLM-Enhanced Clinical Skills Training

Paper Title

"I Wouldn't Really Use It as a Practice Tool": Understanding Medical Students’ Perspectives and Needs on LLM-Enhanced Clinical Skills Training

Publication Info

• Topic area: Medical education and Human-Computer Interaction (HCI) using Large Language Models (LLMs).
• Keywords: Medical education, clinical skills training, Large Language Models, Type 2 Diabetes, user-centered design, co-design workshops, simulated patients, adaptive learning, multimodal outputs, evidence-based medicine.

Background and Problem

• Problem / challenge: Current LLM-based tools for medical education focus on system performance and educational outcomes but overlook learners’ authentic experiences and specific needs. This gap risks poor adoption and suboptimal learning outcomes.
• Significance: Addressing this gap is critical for improving clinical skills training, which is essential for preparing medical students to deliver high-quality healthcare, particularly in complex and high-stakes environments.
• Motivation and related work: Traditional methods like standardized patients and internships are resource-constrained and fail to capture real-world complexity. Prior HCI research explored technologies like VR and simulators but faced limitations in adaptability and realism. LLMs offer potential for dynamic, personalized learning experiences, yet their application in clinical education remains underexplored, particularly from a user experience perspective.

Solution

• Proposed approach: A two-phase study combining probe-based investigations and co-design workshops to identify challenges and expectations of medical students using LLM-based tools for Type 2 Diabetes (T2D) clinical skills training.
• Novelty:
  1. Identification of 11 key challenges across dialogue content, presentation, and interaction dimensions.
  2. Elicitation of medical students’ expectations for enhanced realism, personalization, multimodal outputs, and collaborative features.
  3. Proposal of ten interconnected design considerations for LLM-based medical education tools.
• Procedure and key techniques:
  • Phase 1: Probe-based study with 14 medical students using five T2D-focused LLM probes (e.g., simulated patients, diagnostic decision-making, patient education).
  • Phase 2: Co-design workshops to collaboratively develop expectations and design improvements for LLM tools.
  • Data collection included Think-Aloud verbalizations, semi-structured interviews, and participant-generated artifacts like sketches and annotations.
  • Thematic analysis was conducted to identify recurring challenges and expectations.

Results

• Concrete findings:
  • Identified 11 challenges, including lack of realism in simulated patients, information overload, limited multimodal outputs, and absence of feedback.
  • Proposed ten design considerations, such as enriched patient models, structured content presentation, multimodal outputs, and collaborative mechanisms.
• Advantage over baselines:
  • Shifted focus from system performance to learner experience, addressing usability barriers and aligning tool design with medical students’ needs.
• Experiments / evaluation:
  • Probe-based study with 14 participants (22–25 years old, 8 female, 6 male) using five T2D-related scenarios.
  • Co-design workshops generated 14 prototype sketches and 56 annotated post-it notes, revealing expectations for improved realism, personalization, and interactivity.
• Limitations and future work:
  • Limited participant diversity (Chinese universities, T2D focus).
  • Short study duration; long-term usage patterns remain unexamined.
  • Interaction with researcher-designed prompts may not fully reflect naturalistic use.
  • Future work should expand to other clinical domains, deploy functional systems, and investigate longitudinal effects.

Summary

This study explores medical students’ challenges and expectations when using LLM-based tools for Type 2 Diabetes clinical skills training. Through a two-phase approach, researchers identified usability barriers across dialogue content, presentation, and interaction dimensions and elicited user-driven design considerations. Findings emphasize the importance of learner-centered design, proposing enhancements like enriched patient models, multimodal outputs, and collaborative features. These insights contribute to developing adaptable, impactful LLM-powered tools for medical education, with potential applications beyond T2D and across diverse clinical contexts.