eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR

Research Background and Problem Statement

• Problem and Challenges: Learning Point-of-care Ultrasound (PoCUS) requires complex visuomotor coordination and troubleshooting skills, yet trainees often lack adequate guidance resources such as high-fidelity simulators or experienced mentors. Existing Mixed Reality (MR) tutorial systems typically focus on fixed-action tasks and fail to explain "what the problem is" and "why specific actions are needed" in open-ended tasks.
• Significance: PoCUS has become a critical skill in modern healthcare, playing a vital role in bedside rapid diagnosis and treatment. However, trainees often struggle to integrate theoretical knowledge with practice and frequently feel confused when dealing with complex anatomical structures or tasks requiring high precision.
• Motivation and Related Work: Although high-fidelity simulators and various MR teaching systems exist, most focus on "how to perform actions" without delving into why certain actions are taken or how to adapt to different situations. For instance, existing systems often use arrows or paths to guide fixed tasks but lack support for solving open-ended problems.

Proposed Solution

• Proposed Approach:
  • eXplainMR System: A Mixed Reality teaching system capable of automatically generating textual and visual explanations, focusing on deconstructing complex tasks (e.g., cardiac PoCUS) through scaffolding subgoals.
  • Core Components:
    1. Subgoal Generation: Breaking down complex operations into step-by-step goals to help trainees gradually understand and apply the tasks.
    2. Textual Explanations: Generating detailed anatomical and problem descriptions for each step to explain the "why."
    3. Real-time Image Segmentation and Feedback: Comparing current imaging with target imaging in real-time, using annotations to help trainees identify errors and omissions (e.g., marking imaging issues with "×" and "?" symbols).
    4. 3D Animation Prompts: Providing dynamic views of cross-sectional heart anatomy and target structures, combined with animated "slice movement paths" to guide users in making correct adjustments.
• Innovations:
  1. Integrating anatomical explanations with operational guidance to foster deeper cognitive engagement during skill acquisition.
  2. Automatically generated feedback using 2D-3D interactions to help trainees build visuospatial models.
  3. Enriched problem design (e.g., real-time error annotations and subgoal optimization) to focus trainees on problem-solving rather than merely following instructions.
• Implementation Steps:
  1. Subgoal Generation Logic: Using optimization algorithms to iteratively generate the next suggested step based on the similarity between current and target imaging.
  2. Multi-level Feedback Generation:
     • Using a 3D heart model of anatomical structures to form subgoal segmentation and explanations.
     • Generating annotated images and 3D animations by comparing current and target views.
  3. User Interaction: Trainees operate a virtual probe via head-mounted devices, following system prompts to learn and adjust.

Research Outcomes

• Specific Results:
  • eXplainMR improves understanding of anatomical knowledge and systematic problem-solving compared to traditional arrow or shadow-based visual guidance.
  • User studies show that trainees using eXplainMR reduced the time required to complete PLAX (Parasternal Long Axis View) by 73.1%, compared to a 55.4% reduction with traditional methods.
  • The subgoal generation method received positive evaluations from domain experts, with its optimization steps deemed more aligned with real-world teaching or training scenarios.
• Advantages:
  1. Enhances trainees' understanding of anatomical structures and systematic thinking in probe adjustments.
  2. Reduces cognitive load by breaking problems into subgoals, increasing operational accuracy.
  3. Provides a low-cost, self-directed training environment, supporting education even in the absence of mentors.
• Experimental or Evaluation Results:
  • eXplainMR outperforms in learning complex tasks requiring high cognitive demands (e.g., fine-tuning angles, anatomical recognition).
  • Qualitative feedback indicates that trainees find the system's 3D visual prompts and real-time image annotations more intuitive in explaining the causal relationships behind movement adjustments.
• Limitations and Future Directions:
  1. Limitations:
     • The current MR control simulator lacks the tactile feedback of a real probe, potentially affecting real-world skill transfer.
     • Evaluations focus primarily on short-term behavioral improvements, with limited investigation into long-term skill retention or application in real clinical scenarios.
     • While subgoals benefit beginners, they may hinder experienced trainees from performing tasks more fluidly.
  2. Future Directions:
     • Expanding the system to teach ultrasound for other complex organs and exploring the generalizability of the automated subgoal generation method in broader medical domains.
     • Conducting in-depth studies on the real-world transferability of these MR-generated skills in clinical settings.
     • Enhancing user interface and interaction design, such as incorporating adaptive feedback to cater to trainees with varying levels of experience.

By integrating innovations in cognitive science and visual technology, eXplainMR offers an efficient and in-depth approach to learning cardiac ultrasound, providing a novel perspective and feasible solution to key challenges in medical training.