ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring

Title of the Paper

ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring

Paper Information

• Subject Area: Applications of Artificial Intelligence and Mixed Reality in Telemedicine
• Keywords: Mixed Reality, Augmented Reality, Virtual Reality, Surgery, Collaboration, Telementoring, System Design, Medical Education, Real-Time Technology, Deep Learning

Research Background and Issues

• Problems and Challenges:

  • Traumatic surgeries often require timely intervention, but access to life-saving expertise is not always guaranteed at the patient's location.
  • Existing telemedicine systems are limited to video annotations, supporting only basic interaction forms, which are insufficient for teaching and guiding complex surgical scenarios.
  • The lack of efficient mixed reality technologies for cross-regional collaboration leads to operational inefficiencies and increased potential for guidance errors.

• Significance:

  • Rapid trauma response can save lives, making the efficiency of remote guidance critically important.
  • Providing effective collaborative tools for on-site doctors or technicians in resource-limited or inexperienced settings can significantly improve the quality of medical services.

• Research Motivation and Related Work:

  • The authors have conducted extensive research in the field of mixed reality, including visual communication technologies for remote collaborative tasks, interface evaluation, and gesture technology applications.
  • They identified design limitations in remote collaboration technologies for surgical environments. Existing systems, such as STAR (Surgical Telementoring using Augmented Reality), have demonstrated potential but still lack advanced annotation capabilities and scalability for complex operations.

Proposed Solution

• Methodology and Solution:

  • Proposed a mixed reality system combining virtual reality (VR) and augmented reality (AR), named ARTEMIS.
  • ARTEMIS enables remote experts to interact with 3D reconstructions of internal organs and collaborate with local doctors to perform complex surgeries.
  • The system consists of two core components: the expert interface (ARTEMIS VR) and the novice interface (ARTEMIS AR), integrating camera devices, depth sensors, and movable hardware.

• Innovative Features:

  • ARTEMIS is the first system to provide gesture-based, 3D annotation and real-action guidance in real-time virtual environments.
  • Combines expert gestures, 3D patient models, and reference videos to enhance the interactivity of remote guidance.
  • Optimized interface design from a human-computer interaction perspective, reducing the cognitive load on novices during surgical procedures.

• Implementation Steps and Technologies:

  • Used role-playing and iterative design methods to plan user-friendly system functionalities based on medical professionals' habits.
  • Incorporated hardware such as depth cameras (Azure Kinect, Intel RealSense), OptiTrack optical tracking systems, and Microsoft HoloLens to enable AR/VR interactions.
  • Developed annotation tools (3D pen), real-time streaming capabilities, and patient model reconstruction using the Unity engine.

Research Outcomes

• Specific Results:

  • Successfully developed the ARTEMIS system and evaluated it through 22 surgical procedures (including simulations on human models and cadavers).
  • Preliminary results demonstrate that the system enables inexperienced operators to complete complex surgeries under expert guidance, significantly improving time efficiency.

• Advantages Over Existing Solutions:

  • Overcomes the limitations of traditional 2D annotations by enabling more precise body position marking through 3D depth annotations.
  • Mixed reality interfaces are better suited for medical scenarios, integrating expert gestures with video-based instructional guidance.
  • Supports asynchronous video clip transmission as an additional resource.

• Experimental or Evaluation Results:

  • Preliminary qualitative evaluations revealed that ARTEMIS performs well in terms of accuracy, communication clarity, and interactive guidance, with both experts and novices expressing satisfaction with the system.
  • Addresses issues such as poor video clarity, misaligned AR annotations, and insufficient depth perception by providing richer views through multi-angle cameras.

• Limitations and Future Directions:

  • Current depth cameras have limitations in capturing fine details (e.g., blood vessels or tissues).
  • Depth perception and alignment in VR annotations require further optimization; visual aids or constraints may improve user experience.
  • The system design primarily focuses on the expert perspective; future plans include understanding the specific needs of novices to enhance the interface.
  • Plans to conduct formal comparative studies with other remote collaboration systems and extend the technology to mobile platforms.

This study highlights the potential of mixed reality technologies in the medical field, particularly in remote guidance and education. However, addressing technical limitations and gaining deeper insights into user needs will be the focus of future work.