PinchCatcher: Enabling Multi-selection for Gaze+Pinch

Research Background and Issues

• What problems or challenges did the authors identify?

  • In extended reality (XR) eye-tracking and gesture interaction, multi-item selection functionality remains underdeveloped, particularly in terms of effectively supporting multi-item selection without significantly increasing user burden. Although gaze+pinch interaction holds potential in XR, challenges such as limitations in hand posture shaping (e.g., pre-shaping), competition for gaze attention, and insufficient throughput for multi-item selection persist. Additionally, traditional multi-item selection methods need to be adapted for XR environments.
  • Current multi-item selection techniques, especially those relying on persistent modes or complex gesture switching, often introduce higher cognitive or physical burdens.

• Why is this problem important?

  • Multi-item selection is widely used in user interface operations (e.g., file management, gaming), and its efficiency and usability directly impact user experience. Therefore, designing efficient and intuitive multi-item selection methods in complex XR environments is crucial.
  • Modern XR headsets (e.g., HoloLens 2, Quest Pro) are becoming increasingly popular. These devices integrate eye-tracking and gesture recognition capabilities, providing a foundation for innovative interaction models.

• Research Motivation and Related Work

  • Current research primarily focuses on multi-item selection design for 2D desktop or touchscreen devices, with limited exploration of multi-item selection techniques combining eye-tracking and gestures.
  • While some studies have investigated the combination of gestures and eye-tracking, efficient designs for multi-mode transitions remain underdeveloped. For instance, multi-item selection methods relying solely on eye-tracking or incorporating additional complex gestures address some issues but often increase user burden.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a multi-item selection scheme called "PinchCatcher," which combines eye-tracking with semi-pinch gestures and designs four trigger mechanisms for sub-selection:
    1. SemiDwell: Triggering selection based on gaze duration.
    2. SemiSwipe: Triggering selection through leftward hand swiping.
    3. SemiTilt: Triggering selection through rightward hand tilting.
    4. SemiNDH: Triggering selection via pinching with the non-dominant hand.
  • These methods allow users to enter a "quasi-mode" using the semi-pinch state and complete flexible multi-item selection before performing the full-pinch gesture.

• What are the innovative aspects of this solution?

  1. Using semi-pinch gestures as a mode-switching tool, combined with eye-tracking, the authors designed a natural and XR-compatible modal switching mechanism for multi-target manipulation.
  2. Introducing a single-handed interaction mode, enabling users to focus less on hand positioning and allowing the other hand to perform auxiliary tasks (e.g., scrolling through lists, adjusting objects).
  3. Proposing multiple trigger logics for sub-selection and conducting detailed comparisons, balancing simplicity, low cognitive load, and clear intent.

• What are the implementation steps? What key technologies were used?

  1. Entering the multi-item selection mode through semi-pinch gestures (controlling the distance between fingers).
  2. Using eye-tracking and the four aforementioned methods to trigger specific sub-selection.
  3. Completing multi-item selection and initiating corresponding operations with a full-pinch gesture.
  4. Implementing the experimental system on Meta Quest Pro, including semi-pinch detection, gaze point detection, and sub-selection trigger logic.

Research Outcomes

• What specific results were achieved?

  • Task Completion Time: There was no significant difference in task completion time across different techniques, but an increase in the number of targets significantly extended completion time.
  • Error Rate: SemiSwipe significantly reduced error rates, and its overall selection stability outperformed all other modes.
  • User Evaluation: SemiSwipe and SemiDwell were the most favored methods among participants, while SemiNDH received lower ratings due to its high physical burden.
  • User Experience: SemiSwipe received high overall usability ratings, with the best evaluation scores attributed to its trigger mechanism being less strongly tied to gaze.

• What advantages does this solution have compared to existing ones?

  • Compared to traditional multi-item selection methods based on fixed modes or complex gesture segmentation, PinchCatcher simplifies user operations and reduces cognitive load.
  • The semi-pinch + eye gaze combination effectively avoids the high learning curve and user fatigue issues present in current solutions.

• What were the experimental or evaluation results?

  • Semi-pinch technique outperformed two-handed methods: While FullDH (two-handed technique) was more familiar to users, it exhibited disadvantages in terms of error rate and fatigue.
  • Outstanding advantages of SemiSwipe: Low error rate, positive interaction evaluation, and ease of use; however, its large hand movement amplitude suggests room for optimization.
  • Polarized evaluation of SemiDwell: Low physical burden but prone to accidental activation (Midas Touch).
  • Suboptimal performance of SemiTilt: Issues with sensitivity and fatigue.

• Limitations and Future Directions

  1. Scope limitations: The current study only explores sequential multi-item selection for small target sets, leaving large-scale target selection and parallel selection techniques to be further developed.
  2. Hardware dependency and setup constraints: The experiments relied on specific hardware and thresholds, necessitating broader adaptability and personalized settings in the future.
  3. Extension to more scenarios: Validation in other complex 3D scenarios (e.g., dynamic objects, occlusions) is needed to assess the technology's applicability.
  4. Improvement of feedback mechanisms: Exploring more forms of visual, auditory, or haptic feedback could further enhance user awareness and accuracy.