CaseMaster: Designing and Evaluating a Probe for Oral Case Presentation Training with LLM Assistance

Paper Title

CaseMaster: Designing and Evaluating a Probe for Oral Case Presentation Training with LLM Assistance

Publication Info

• Topic area: Enhancing oral case presentation (OCP) training in medical education using large language models (LLMs).
• Keywords: Oral case presentation, medical education, large language models, training systems, interactive learning, SOAP format, feedback integration, adaptive learning, educational technology, user-centered design.

Background and Problem

• Problem / challenge: Traditional methods for training oral case presentations (OCPs) face challenges such as uneven student engagement, lack of structured guidance, limited access to timely feedback, and high resource demands. Existing automated feedback systems focus on public speaking but fail to address the specific needs of OCPs.
• Significance: OCPs are critical for medical communication, enabling clear and efficient information transfer among healthcare professionals. Improving OCP training can enhance diagnostic reasoning, presentation clarity, and confidence in medical students.
• Motivation and related work: Prior research has explored frameworks like SOAP and tools for presentation training but lacks comprehensive models tailored to OCPs. LLMs offer potential for adaptive, personalized learning, but their integration into medical education remains underexplored. This study builds on these gaps by designing and evaluating an LLM-assisted training system.

Solution

• Proposed approach: CaseMaster, an interactive system probe leveraging LLMs to assist medical students in OCP training through structured preparation and reflection stages.
• Novelty:
  1. Development of a two-stage system (Preparation and Reflection) tailored to OCP training.
  2. Integration of LLM-generated content for structured guidance, feedback, and solution comparison.
  3. Proposal of design guidelines for LLM-based educational tools in medical training.
• Procedure and key techniques:
  • Preparation Stage: Users explore patient cases, draft presentations using the SOAP format, and receive LLM-assisted guidance for content refinement.
  • Reflection Stage: Users compare their solutions with reference answers, receive LLM-generated feedback, and evaluate their performance using a detailed score sheet.
  • Implementation includes modular prompt templates, temperature tuning for LLM outputs, and validation of LLM-generated content by medical educators.

Results

• Concrete findings:
  • CaseMaster improved clarity in differential diagnoses (M = 5.00 vs. baseline M = 4.00, padj = 0.042).
  • High inter-rater reliability for LLM-generated scoring (ICC = 0.88).
  • Participants rated CaseMaster highly for system satisfaction (M = 6.00, IQR = 0.25).
• Advantage over baselines:
  • Enhanced presentation quality, particularly in reasoning and organization.
  • Reduced cognitive load due to structured guidance and feedback.
  • Higher trust and usability ratings compared to traditional tools.
• Experiments / evaluation:
  • Controlled study with 12 medical students comparing CaseMaster to a baseline system.
  • Expert evaluation with five medical educators assessing system reliability and educational impact.
  • Metrics included presentation quality, task workload (NASA-TLX), and usability (SUS).
• Limitations and future work:
  • Limited to orthopedic cases; future work will expand to other medical specialties.
  • Small sample size (12 students, 5 educators); larger-scale studies needed.
  • Lack of pre- and post-intervention skill assessments.
  • Opportunities for integrating multimodal formats (e.g., flowcharts, videos) and customizable prompts.

Summary

CaseMaster is an LLM-assisted system designed to enhance OCP training for medical students through structured preparation and reflection stages. The system demonstrated significant improvements in presentation quality, particularly in differential diagnoses, and reduced cognitive workload compared to traditional methods. Expert evaluations highlighted its potential to streamline feedback and support evidence-based critical thinking. While the study focused on orthopedic cases, future work aims to expand its applicability across medical disciplines and incorporate multimodal and customizable features for broader educational impact.