iText: Hands-free Text Entry on an Imaginary Keyboard for Augmented Reality Systems

Title of the Paper

iText: A Hands-Free Text Input Technique for Virtual Keyboards in Augmented Reality Systems

Bibliographic Information

• Research Domain: Human-Computer Interaction and Text Input Techniques in Augmented Reality (AR)
• Keywords: Text input, hands-free input, eye tracking, dwell time, augmented reality, head-mounted display, virtual keyboard

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Existing text input techniques for AR head-mounted display (HMD) devices often rely on mid-air gestures, which can lead to arm fatigue.
  • Small transparent screens cause visual obstruction, preventing users from clearly observing other virtual or real objects in the environment.
  • Current text input techniques based on mid-air visible keyboards have limited applicability in daily use scenarios, lacking natural operability.

• Importance of the Problem:

  • Efficient and convenient text input is crucial for the widespread adoption of AR systems. Particularly in human-computer interaction, a solution is needed that addresses both arm fatigue and environmental visibility.

• Research Motivation and Related Work:

  • Text input is a critical function in AR applications, but existing technologies have not adequately addressed hands-free input and field-of-view obstruction issues.
  • Related studies have shown that virtual keyboards (e.g., "invisible" keyboards) and eye-tracking input can be efficient and reliable under specific conditions, but they have yet to be systematically applied to AR devices.

Proposed Solution

• Solution Overview:

  • Developed a text input technique named iText, a hands-free input method based on a "virtual (invisible) keyboard" designed for AR HMD devices.
  • Introduced three different input mechanisms: Eye-based E-Type (Eye Tracking), Dwell-based D-Type (Dwell Time), and Gesture-based G-Type (Head Gestures).

• Innovative Features:

  • Combined virtual keyboards to enable users to accurately recall and select key positions without requiring a visible keyboard, achieving hands-free and unobstructed text input.
  • The system collects user interaction data and employs statistical decoding algorithms for text prediction, further improving input efficiency and accuracy.

• Implementation Steps and Key Technologies:

  1. Virtual Keyboard Design: Displayed a virtual keyboard at a distance of 1 meter within the user’s field of view, measuring 40 x 20 cm, with 30% transparency. Key positions are hidden, but keys like space and delete are marked.
  2. Three Input Mechanisms:
     • E-Type: Users select characters by blinking their eyes, minimizing accidental selections.
     • D-Type: Characters are selected based on a dwell time of 600 milliseconds, with visual circular progress bars and auditory feedback.
     • G-Type: Users draw character gestures using head movements to complete selections.
  3. Statistical Decoder Design: Utilizes statistical models to infer the most likely characters and words selected by users, incorporating a language model to enhance prediction accuracy.

Research Findings

• Specific Results:

  • Users can efficiently input text hands-free using the AR virtual keyboard.
  • Average text input speeds for the three input mechanisms are: E-Type (13.76 WPM in long-term testing), D-Type (9.03 WPM), and G-Type (9.84 WPM).
  • In long-term testing, E-Type achieved an average text error rate of only 1.5%, with low levels of eye fatigue reported by users.

• Advantages Over Existing Solutions:

  • iText addresses both arm fatigue and visual obstruction issues present in current AR text input technologies.
  • Supports efficient text input in scenarios where users’ hands are occupied or in dynamic environments (e.g., walking or attending lectures).

• Experimental and Evaluation Results:

  • The first user study demonstrated that users could successfully recall key positions and perform hands-free input using the virtual keyboard.
  • The second user study evaluated the performance and user experience of the three input techniques, with E-Type significantly outperforming the other two mechanisms.
  • The third long-term study revealed that users, when using E-Type exclusively, could significantly improve input speed within five days while maintaining low error rates and minimal eye fatigue.

• Limitations and Future Directions:

  1. The impact of keyboard size variations on input performance has not been fully assessed; future work could explore testing different keyboard designs.
  2. Current experiments were conducted in controlled laboratory environments, and diverse real-world settings may affect input performance.
  3. The limited field of view of the current device (HoloLens 2) may constrain the virtual keyboard design; enhancing display capabilities or future technological iterations could further optimize the design.

Potential Application Scenarios

• Real-time text input while walking, reducing visual obstruction when users’ hands are occupied.
• Quickly retrieving definitions or annotations during lectures or meetings without disrupting the field of view.
• In VR gaming scenarios, users can reply to messages quickly without interrupting gameplay.