Tangible Immersive Trauma Simulation: Is Mixed Reality the next level of medical skills training?

Document Title

Tangible Immersive Trauma Simulation: Is Mixed Reality the Next Level of Medical Skills Training?

Document Information

• Subject Area: Medical skills training and mixed reality technology
• Keywords: Mixed Reality (MR), medical training, paramedic training, haptic feedback, presence, medical simulation

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Current medical simulation training methods are divided into two domains: skills training using physical mannequins and scenario-based assessment and decision-making training using Virtual Reality (VR). There is minimal overlap between these two approaches.
  • While VR provides high levels of visual and auditory realism, it lacks haptic feedback. On the other hand, physical mannequins offer tactile operability but cannot dynamically respond to environmental and situational changes.
  • Mixed Reality (MR) could be a potential solution that combines the advantages of both, but its application in the medical field remains underdeveloped.

• Research Significance:

  • Paramedics need to make accurate decisions and perform procedures within minutes. Enhancing the immersive and practical aspects of their training can improve performance in real-world scenarios.
  • Introducing MR technology into medical training could enhance trainees' operational and decision-making capabilities in complex scenarios.

• Motivation and Related Work:

  • Existing VR and haptic devices have been used in general learning environments as well as in areas such as surgical training and basic life support, but these implementations are largely independent.
  • A few studies have attempted to combine physical simulation with virtual augmentation, such as enhancing physical mannequins with virtual overlays. Building on this foundation, the authors explore whether MR can integrate medical skills and scenario training.

Proposed Solution

• Innovative Approach or Solution:

  • Developed an MR training system that combines physical mannequins with virtual environments. By overlaying virtual visual elements (e.g., dynamic changes in patient conditions), trainees can use real medical tools to interact with physical mannequins.
  • The system incorporates olfactory, auditory (e.g., ambient noise and patient breathing sounds), tactile, and visual enhancements (e.g., feedback such as cyanosis of the patient’s skin) to increase immersion and realism.

• Implementation Steps and Key Technologies:

  1. Technical Architecture:
     • Utilized the Varjo XR-3 headset for real-time video overlay and virtual-physical scene tracking.
     • Employed green screen technology to overlay virtual patient visuals, with LiDAR sensors capturing depth information of hands and tools.
     • Equipped medical devices (e.g., airway bags, oxygen tanks) with haptic feedback.
  2. Scenario Design:
     • Created a typical car accident scenario involving two virtual patients (Grace and Tobi), guiding trainees through initial assessment, emergency treatment, and patient handover.
  3. User Interaction:
     • Users can freely move within the scenario, using real medical tools (e.g., stethoscopes, ventilators) to examine patient conditions and trigger virtual feedback (e.g., chest rise and fall).
     • Medical experts role-play as patients (via voice acting) to enhance social interaction during training.

Research Outcomes

• Specific Results:

  • Experimental results showed that participants experienced high levels of physical and self-presence in the MR scenario and perceived significant stress levels (validated through skin conductance measurements).
  • User questionnaire data indicated high acceptance of MR training technology and its intended effects, with particularly high scores for immersion and intuitive interaction.

• Advantages Over Existing Solutions:

  • Enhanced the tactile and realism aspects of existing VR training, bridging the gap between virtual and physical skills training.
  • Improved scenario credibility through dynamic simulation of patient conditions (e.g., breathing, skin color changes).
  • Increased trainees' intuitiveness and engagement by using real tools.

• Experimental or Evaluation Results:

  • Technology Acceptance: High scores for metrics such as enjoyment and ease of use (most scores above 4.5 out of 5).
  • Stress and Arousal Levels: Compared to baseline quiet states, trainees exhibited significantly higher skin conductance peak frequency and amplitude during the scenario.
  • Presence: While physical and self-presence were high, social presence was slightly lower, indicating room for improvement in virtual character social interaction.

• Limitations and Future Directions:

  • Limitations:

    • The physical mannequin in the scenario only included the torso (no limbs), limiting the scope of operations.
    • Transparent tools (e.g., oxygen masks) had display issues in the green screen environment.
    • Limited facial expressions and interactive behaviors of virtual agents affected social presence.
    • Small sample size with only preliminary evaluations conducted; comparative studies with traditional training methods are yet to be performed.

  • Future Directions:

    • Integrate more complete and mobile physical mannequins, adding fine-grained feedback such as skin texture and pulse.
    • Leverage AI technologies to enhance the interactive responsiveness of virtual agents.
    • Expand the tool library to cover all equipment used in medical scenarios, enabling simulation of more complex operational decisions.
    • Conduct randomized controlled trials to quantitatively evaluate the advantages of MR in medical training and study its long-term learning outcomes.

Conclusion

The authors proposed and validated a novel MR-based skills training solution that combines virtual and physical interactions. The study demonstrated its potential for medical emergency training and identified key areas for future optimization.