Interaction Through Instruments: Extending Surgical Instruments as Interaction Devices

Paper Title

Interaction Through Instruments: Extending Surgical Instruments as Interaction Devices

Publication Info

• Topic area: Human-Computer Interaction in surgical contexts
• Keywords: Interaction Through Instruments, Minimally-Invasive Surgery, surgical instruments, autonomy, focus shifts, mental effort, intraoperative needs, visualization layouts, touchless interaction, participatory design

Background and Problem

• Problem / challenge: Current touchless interaction techniques in surgery, such as mid-air gestures, gaze-based systems, and voice commands, redirect attention away from the surgical field, disrupt workflows, and fail to fully address sterility constraints. Delegating interaction to assistants introduces delays, miscommunication, and loss of autonomy.
• Significance: Autonomous interaction is critical for surgeons to maintain focus, reduce delays, and make informed decisions during surgery without compromising sterility or workflow efficiency.
• Motivation and related work: Prior work has explored touchless interaction and repurposing objects for interaction but has not addressed the specific needs of domain experts using task instruments. This paper builds on these gaps by proposing a paradigm that leverages surgeons’ embodied knowledge of their instruments for interaction.

Solution

• Proposed approach: Interaction Through Instruments (ITI), a paradigm where surgical instruments are repurposed as interaction devices to control digital information systems without modifying the instruments.
• Novelty:
  1. Introduction of the ITI concept, enabling interaction without disrupting sterility or primary tasks.
  2. Empirical understanding of intraoperative needs and challenges through surveys and workshops.
  3. Development of InteractOR, a system enabling interaction via surgical instruments with two visualization layouts: overlay and side-by-side.
  4. Evaluation of ITI’s impact on autonomy, efficiency, and focus shifts compared to delegation.
• Procedure and key techniques:
  • Conducted a survey (N=23) to identify intraoperative needs and challenges.
  • Held participatory design workshops (N=10) to design interaction gestures and interface layouts.
  • Implemented InteractOR, using instrument segmentation and pinch-gesture recognition for interaction.
  • Evaluated InteractOR through Comparative Structured Observation (N=12), comparing ITI to delegation and assessing tradeoffs between overlay and side-by-side layouts.

Results

• Concrete findings:
  • ITI reduced focus shifts (Delegation: Med=5; Overlay/Side-by-Side: Med=3), mental effort (Delegation: Med=7; Side-by-Side: Med=4.5; Overlay: Med=6), and time to lesion identification (Delegation: M=120.9s; Overlay: M=58.9s; Side-by-Side: M=53.7s).
  • Both ITI layouts provided high autonomy ratings (Med=5), significantly outperforming delegation (Med=1).
• Advantage over baselines: ITI outperformed delegation by maintaining autonomy, reducing delays, and improving workflow efficiency. Overlay minimized gaze shifts but introduced occlusion-related effort, while Side-by-Side reduced effort but required more gaze shifts.
• Experiments / evaluation:
  • Participants performed lesion identification tasks under three conditions: delegation, overlay, and side-by-side.
  • Data collected included focus shifts, mental effort, efficiency, exploration behaviors, and perceived autonomy and safety.
  • Mixed-method analysis combined qualitative interviews and quantitative measures (Likert scales, Paas scale, time logs).
• Limitations and future work:
  • Limited gestures and single-instrument interaction in the prototype.
  • Controlled lab environment may underestimate real-world delegation costs.
  • Future work should explore multi-instrument setups, gesture vocabularies, and real-world surgical contexts.

Summary

This paper introduces Interaction Through Instruments (ITI), a paradigm repurposing surgical instruments for interaction with digital systems, addressing sterility constraints and workflow disruptions. Through surveys, workshops, and a system implementation (InteractOR), ITI was shown to reduce focus shifts, mental effort, and delays compared to delegation, while providing surgeons with greater autonomy. Two visualization layouts—overlay and side-by-side—were evaluated, revealing tradeoffs between occlusion and gaze shifts. The findings highlight ITI’s potential to enhance intraoperative interaction and suggest broader applications in other domains requiring bimanual physical tasks.