From Concept to Clinic: Multidisciplinary Design, Development, and Clinical Validation of Augmented Reality-Assisted Open Pancreatic Surgery

Research Background and Issues

• What problems or challenges did the authors identify?

  • The authors pointed out that although augmented reality (AR) technology has shown great potential in surgical navigation, the development of AR systems for open abdominal surgeries still faces numerous limitations. Specific challenges include organ displacement and deformation, environmental factors (such as time pressure, hygiene requirements, and precision demands), and tracking difficulties caused by occlusion and lighting conditions. Additionally, most AR surgical assistance systems are primarily focused on applications using tablets, portable devices, or monitors, rather than fully utilizing wearable AR technology.

• Why is this issue important?

  • Abdominal surgeries, particularly pancreatic tumor resection, are high-risk due to the pancreas's location and the complex anatomical structure of nearby critical blood vessels. Precisely locating the positions of blood vessels and tumors is crucial to minimizing damage to surrounding healthy tissues.

• Research Motivation and Related Work

  • This study aims to fill the gap in AR technology validation for open surgeries and develop an AR-assisted system (ARAS) for pancreatic surgeries. Related work indicates that the application of AR technology in surgeries has largely been confined to laboratory environments or in vitro testing, lacking validation in real surgical settings. Furthermore, existing AR-based surgical navigation systems (such as monitor- or handheld device-based systems) provide accurate information but have limited clinical effectiveness due to distracting surgeons from focusing on surgical operations.

Solution

• What methods or solutions did the authors propose?

  • The authors developed a user-driven AR assistance system—ARAS, which provides real-time navigation support for open pancreatic surgeries using wearable AR technology (HoloLens 2).
  • ARAS integrates a dual-layer marker-based registration method to ensure accurate registration during critical surgical stages. In the early stage, markers are positioned on the sternum to provide stable and continuous attachment points; in the later stage, auxiliary markers are placed on the vena cava branches.

• What are the innovative aspects of this solution?

  • The dual-layer marker registration method addresses the issue of registration inaccuracies caused by organ movement during surgery.
  • A fully wearable AR system was developed specifically for pancreatic surgeries, integrating preoperative data into real-time surgical navigation.
  • A fully voice-controlled interaction model was implemented to overcome challenges posed by limited space and hygiene requirements.

• What are the implementation steps and key technologies used?

  • Development and Implementation:
    • The ARAS software was designed using the Unity 3D game engine, compatible with HoloLens 2, providing patient-specific anatomical models and related data visualization.
    • 3D models of critical blood vessels and tumors near the pancreas were reconstructed using CT scans.
    • During the preoperative phase, reliable registration was achieved through dual-layer markers (sternum and vena cava branch markers).
  • Clinical Validation:
    • 20 clinical trials were conducted in highly complex pancreatic resection surgeries, with system accuracy validated through ultrasound and surgeons' subjective assessments.

Research Outcomes

• What specific outcomes were achieved?

  • The ARAS system successfully enabled accurate visualization of patient-specific data during pancreatic surgeries and optimized preoperative and intraoperative decision-making.
  • Clinical trial results showed high accuracy during the early stages of surgery (preparation and resection phases), with an average registration quality score of approximately 4.6/5.

• What advantages does it have compared to existing solutions?

  • Compared to monitor- or handheld device-based AR systems, ARAS's wearable design eliminates the need for surgeons to divert their gaze, improving workflow efficiency.
  • The dual-layer marker method significantly enhanced registration accuracy, making it suitable for complex surgical environments.

• What were the experimental or evaluation results?

  • The average surgery duration was 245.9 minutes, with ARAS usage accounting for 50.37%. The system quality scored 6.5 out of 7 on the UMUX-Lite user experience questionnaire, corresponding to a SUS score of 82.48 (close to excellent).
  • Postoperative surveys revealed that most surgeons reported avoiding misplacement of blood vessels and tumors through the AR navigation system, improving overall surgical efficiency.

• What are the limitations and future directions?

  • Limitations:
    • Registration accuracy of the 3D model declined in later stages due to organ deformation and movement during surgery.
    • Real-time patient data (such as real-time ultrasound or MRI) was not utilized to improve model accuracy.
    • Long-term use of the HoloLens 2 presented comfort issues.
  • Future Directions:
    • Investigate solutions for integrating real-time organ data updates.
    • Explore methods to enhance the comfort of wearable devices, such as addressing compatibility with magnifying glasses.
    • Develop customized modes tailored to different surgical roles and tasks (e.g., navigation mode vs. surgeon-led mode).

This study demonstrates the potential of AR technology in complex surgeries and provides important design guidance for future integration of AR technology into surgical environments.