STAR: Smartphone-analogous Typing in Augmented Reality

Title of the Paper

STAR: Smartphone-analogous Typing in Augmented Reality

Paper Information

• Domain: Text input technologies in Augmented Reality (AR)
• Keywords: Augmented Reality, text input, smartphone input, head-mounted display, QWERTY keyboard

Research Background and Problem

• Identified Problems or Challenges: While text input is a core and frequent task in Augmented Reality (AR) applications, designing an efficient and user-friendly text input method remains an unresolved challenge. Existing bare-hand AR text input techniques (e.g., mid-air gestures, eye-tracking input, fingertip touch) often lack the tactile feedback provided by physical surfaces, and users are generally unfamiliar with these technologies.
• Significance: With the advancement of AR hardware, providing an efficient text input method for AR environments is crucial for enhancing its application scenarios (e.g., remote work, education, and healthcare). Addressing the aforementioned issues will significantly improve user experience.
• Motivation and Related Work:
  • Two-thumb typing on smartphones is widely familiar to users and relatively fast.
  • Existing technologies suffer from drawbacks such as the lack of tactile feedback and user fatigue.
  • The authors aim to explore how to transfer the two-thumb typing skills of smartphones to a bare-hand input environment in AR.

Solution

• Proposed Method or Solution:
  • The authors propose a novel AR text input technique—STAR (Smartphone-analogous Typing in Augmented Reality), which transfers two-thumb typing skills to the AR environment to address the shortcomings of existing input technologies.
  • Users wear a head-mounted display (HMD) that projects a virtual QWERTY keyboard onto the skin of their hands for input, leveraging the tactile feedback from finger-to-skin contact.
  • The solution allows users to seamlessly switch tasks by using a "knuckle gesture" to activate or deactivate the keyboard.
• Innovations:
  • Leverages users' familiarity with two-thumb typing on smartphones, reducing the learning curve.
  • Optimizes gesture input by enabling finger touch on the skin surface, providing a more natural user experience.
  • High social acceptability due to its resemblance to natural smartphone typing gestures.
• Implementation Steps and Key Technologies:
  • The design phase examined users' natural hand gestures and habits.
  • The implementation phase selected the most suitable skin surface (e.g., the surface of the second finger), keyboard position (fixed-position keyboard), and keyboard size (enlarged keyboard dimensions, key width of 6mm, key spacing of 2mm).
  • Real-time hand tracking and screen projection were achieved using existing AR devices (e.g., Hololens 2).
  • Enhanced feedback, visual depth perception, and other techniques were used to improve user operation stability and input accuracy.

Research Results

• Specific Outcomes:
  • The average input speed of STAR was 21.9 WPM (Words Per Minute), approximately 56% of the speed of smartphone typing (39.4 WPM).
  • The average uncorrected error rate (UER) was 0.3%, significantly lower than the average UER of smartphones (0.5%).
  • Subjective feedback indicated that some users found the STAR experience satisfactory, considering it innovative and feasible.
• Advantages and Evaluation Results:
  • The STAR technique improved text input speed in bare-hand AR environments while maintaining a low error rate.
  • Compared to existing bare-hand input techniques (e.g., eye-tracking input or mid-air gesture input), STAR integrates a familiar input method, reducing the learning curve.
  • Statistical decoding techniques provided spelling suggestions, further enhancing input efficiency.
• Limitations and Future Directions:
  • The current AR devices have limited hand-tracking precision: tracking latency of up to 90 ms and tracking drift of approximately ±1 mm, which affects input accuracy.
  • The keyboard is fixed in a static position, failing to fully emulate the dynamic smartphone typing experience.
  • Fully achieving bare-hand mechanical input requires advancements, such as integrating higher-precision sensor technologies (e.g., ring devices or wristbands).
  • Future research could explore real-world scenario testing to evaluate STAR's applicability and investigate improved designs, such as one-handed input or hand-sliding gesture-based input.