MR Microsurgical Suture Training System with Level-Appropriate Support

Document Title

MR Microsurgical Suture Training System with Level-Appropriate Support

Document Information

• Subject Area: Mixed reality technology and medical training, specifically neurosurgical minimally invasive suture skill training
• Keywords: Neurosurgical training, MR, CV-based tracking, Visual feedback, Real-time feedback

Research Background and Problems

• What problems or challenges did the authors identify?

  1. Minimally invasive suturing in neurosurgery requires exceptional skills, but traditional training methods are inefficient and time-consuming (approximately 10,000 stitches and hundreds of hours).
  2. Privacy and legal issues prevent medical interns from learning directly through observation during surgeries.
  3. Existing virtual reality (VR) training systems have parallax issues and differ significantly from real microscope environments, affecting training outcomes.

• Why is this problem important?

  1. Minimally invasive suturing is a critical skill in neurosurgery, requiring high levels of speed, accuracy, and stability.
  2. The ischemic tolerance time of brain tissue during surgery is extremely short, and surgical efficiency directly impacts patient survival.
  3. Medical training methods need to address privacy, interruption-prone environments, and efficiency to adapt to the busy medical field.

• Research Motivation and Related Work

  1. In recent years, the medical field has increasingly adopted mixed reality (MR) and artificial intelligence (AI) technologies to improve the efficiency of medical skill training.
  2. Previous studies on VR skill acquisition systems have preliminarily validated their effectiveness for individuals without medical backgrounds but lack evaluation and adaptability for the target users—neurosurgeons.
  3. The authors aim to develop a novel mixed reality training system that balances medical professionalism, interruption tolerance, and compatibility with real-world environments.

Solution

Method or Solution

The authors propose a mixed reality (MR)-based minimally invasive suture training system, divided into two training modes based on user skill levels:

1. Novice Mode:

   • Utilizes shadow matching technology to generate virtual "shadows" of expert surgeons' hand movements, helping users practice aligning forceps and needles to ideal positions.
   • Employs deep learning (YOLO v8 algorithm) to segment and recognize surgical tools, assessing training outcomes through image similarity calculations.

2. Intermediate Mode:

   • A real-time feedback system dynamically evaluates speed, accuracy, and stability during different suturing stages ("knot tying," "thread cutting," etc.).
   • Uses voice recognition to segment training stages and provides targeted feedback during user operations (e.g., color changes in forceps movement trajectories, gauze movement warnings).
   • Displays a scorecard after training to quantitatively assess progress, motivating users to further improve their skills.

Innovations of the Solution

• Technical Integration: Combines MR technology with deep learning to achieve markerless recognition of surgical tools, avoiding interference with the surgical experience.
• Feedback Mechanism: Implements real-time feedback through visualization and voice recognition technologies, tailoring training content to different skill levels.
• Multi-Stage Training Modes: Designs specific training tasks for novices and intermediate users, enhancing training efficiency and relevance.

Implementation Steps and Key Technologies

1. Hardware System: Constructs a setup with a microscope camera, video see-through MR headset (Varjo XR-3), computer processor, and graphics card.
2. Software and Algorithms:
   • Uses deep learning algorithms (YOLO v4 and YOLO v8) to recognize tool and gauze movement trajectories.
   • Detects gauze movement using optical flow methods to quantify stability.
   • Employs the SIFT algorithm to calculate image similarity for evaluating novice training outcomes.
3. User Interface Design: Develops targeted visual feedback interfaces, including "shadow matching," "forceps movement circles," "gauze movement warnings," and "scorecards."

Research Results

Specific Outcomes

• Novice Mode Evaluation:

  • Experiments show that the shadow matching system significantly improves novices' tool-handling postures, bringing them closer to ideal states.
  • The system enhances users' ability to engage in prolonged focused training, improving practice efficiency and motivation.

• Intermediate Mode Evaluation:

  • Real-time feedback significantly reduced forceps movement deviations during the "knot tying" stage, improving skill precision.
  • The scorecard, as a gamification feature, effectively boosted user training motivation.

Comparative Advantages Over Existing Solutions

• The MR system reduces the gap between VR training and real-world environments, allowing users to observe both the microscope and the surrounding environment simultaneously.
• Enables real-time quantitative evaluation of complex skills (speed, accuracy, stability) and provides stage-specific guidance.

Experimental or Evaluation Results

• Novice System: Evaluated with 14 participants without medical backgrounds, results demonstrated significant improvements in efficiency and focus.
• Intermediate System: Tested with 6 neurosurgical residents, quantitative data and user feedback supported the system's applicability and effectiveness.

Limitations and Future Directions

• Limitations:

  • The novice experiment duration was relatively short, preventing in-depth validation of the system's long-term training effects.
  • Some users reported physical discomfort due to the headset's weight.

• Future Directions:

  • Design lighter headset equipment.
  • Expand system functionality to support real-time video review training for more advanced surgeons.
  • Extend the technology's application to other medical fields, such as laparoscopic and ophthalmic surgery training.

Supplementary Information

• Funding Information: This research was supported by Japan's JST CREST and JSPS KAKENHI projects.
• Potential Impact: This study has broad implications for improving the acceptability and efficiency of medical training and can be extended to other fields requiring highly precise skills.