SwEYEpinch and Beyond: Exploring Intuitive, Efficient Text Entry for Extended Reality via Eye and Hand Tracking

Paper Title

SwEYEpinch and Beyond: Exploring Intuitive, Efficient Text Entry for Extended Reality via Eye and Hand Tracking

Publication Info

• Topic area: Text entry methods for Extended Reality (XR) using gaze and hand tracking.
• Keywords: Extended Reality, text entry, gaze tracking, hand tracking, pinch gesture, mid-swipe prediction, hybrid input, user studies, XR keyboards, spatiotemporal decoding.

Background and Problem

• Problem / challenge: Text entry in XR is slower and more effortful compared to physical keyboards or touchscreens. Existing methods like controller-raycast, mid-air tapping, and gaze-only techniques face limitations such as fatigue, low speed, and high error rates.
• Significance: Efficient text entry is critical for XR adoption, enabling practical, everyday use of head-worn displays (HWDs) without external peripherals or voice input.
• Motivation and related work: Prior work on gaze typing (e.g., dwell-based methods, gaze swipes) and hybrid approaches (e.g., gaze and hand gestures) has shown potential but remains limited by speed caps, fatigue, and error rates. This paper builds on these methods to address their shortcomings by introducing a hybrid approach that decouples targeting (gaze) from commitment (manual pinch).

Solution

• Proposed approach: SwEYEpinch, a hybrid text-entry method that uses gaze for word tracing and a manual pinch gesture to delimit word input.
• Novelty:
  1. Introduction of mid-swipe prediction, displaying candidate words during gaze swipes.
  2. Implementation of mid-swipe deletion and a deletion peek window for error correction.
  3. Optimized spatiotemporal decoding algorithm (Gaze2Word) for real-time feedback and candidate ranking.
  4. Demonstration of sustained learning and everyday typing speeds through longitudinal user studies.
• Procedure and key techniques:
  • Users gaze at the initial letter of a word, pinch to start the swipe, and trace the word with their gaze.
  • Mid-swipe predictions appear dynamically, allowing users to confirm the word early.
  • The Gaze2Word algorithm processes gaze traces using fixation detection, clustering, and spatiotemporal dynamic time warping (DTW).
  • Error correction is facilitated by mid-swipe deletion and a deletion peek window.

Results

• Concrete findings:
  • SwEYEpinch achieved speeds up to 64.7 WPM in a longitudinal study, with five participants exceeding 54 WPM.
  • Mid-swipe prediction improved speed (22.3 WPM vs. 16.6 WPM for the basic version) without increasing error rates.
  • Participants showed sustained learning over 30 sessions, with consistent WPM gains and reduced error rates.
• Advantage over baselines:
  • Faster and more preferred than gaze-only methods (SkiMR, GlanceWriter XR) and production-realistic baselines (Finger-Tap with Prediction, Hand-Swipe).
  • Maintained competitive or lower error rates compared to baselines.
• Experiments / evaluation:
  • Four user studies (US1–US4) with 71 participants across different conditions.
  • Metrics included words per minute (WPM), total error rate (TER), and user preference.
  • Longitudinal study (US4) tracked performance over 30 sessions across seven days.
• Limitations and future work:
  • Limited correction mechanisms (e.g., whole-word rollback only immediately after commit).
  • Potential fatigue from prolonged gaze use; future work should explore fatigue metrics and mobile scenarios.
  • Current evaluation focused on transcription tasks; future studies should include open-ended writing and older demographics.

Summary

The paper introduces SwEYEpinch, a hybrid XR text-entry method combining gaze-based word tracing with a manual pinch delimiter. Through four user studies, the method demonstrated faster speeds, higher user preference, and sustained learning compared to existing baselines. Key innovations include mid-swipe prediction, error correction tools, and an optimized spatiotemporal decoding algorithm. The results suggest that SwEYEpinch is a viable, efficient solution for everyday XR text entry, achieving speeds comparable to physical keyboards with routine use. Future work will address broader usability scenarios and fatigue mitigation.