TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking Only

Title of the Paper

TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking Only

Paper Information

• Domain: Touch interaction technology in Augmented Reality (AR)
• Keywords: Augmented Reality, touch input, passive surfaces, indirect input, gesture recognition, energy optimization, user experience, interaction design

Research Background and Problem

• Problems and Challenges:

  1. Current mid-air bare-hand input in augmented reality lacks tactile feedback, leading to insufficient precision, user fatigue, and poor experience.
  2. Existing touch solutions (e.g., passive surface touch in AR) require additional instrumentation of the environment or user devices, resulting in high complexity or low adaptability.
  3. Compared to powerful computing devices, AR glasses typically have limited power budgets, imposing constraints on algorithm performance and energy consumption.

• Significance:
  Effectively utilizing uninstrumented ordinary surfaces for touch input can provide a more universally applicable, energy-efficient, and effective AR interaction method, enhancing user comfort and precision.

• Motivation and Related Work:

  • Previous research has shown that physical surfaces can improve the precision and user experience of augmented interactions, but there is a lack of technology that comprehensively addresses all the above limitations. Existing technologies (e.g., MRTouch, OmniTouch) have addressed some issues but still face limitations in diversity, instrumentation requirements, and support for indirect input.

Solution

• Method and Core Technology:
  The authors propose an AR touch technology called TriPad, which requires no additional sensors or environmental instrumentation. It relies on hand tracking data and explicit user gestures to define and operate touch planes.

  1. Touch Plane Creation: Users place their thumb, middle finger, and pinky on the target surface and perform a quick tapping gesture. The system generates a touch plane using the 3D positioning data of the fingers.
  2. Input Modes:
     • Direct Input Mode: Direct interaction aligned with the virtual interface.
     • Indirect Input Mode: Users control the pointer position via a virtual touchpad, enabling interaction with distant or misaligned content.
  3. Device Efficiency: TriPad utilizes only built-in hand tracking APIs and minimal geometric computations, reducing computational overhead.

• Innovations:

  • Eliminates the need for environmental instrumentation, making it adaptable to various ordinary surfaces (e.g., glass, wood, drywall).
  • Operates without additional energy consumption on power-constrained devices like AR glasses, optimizing computational power usage.
  • Provides precise indirect input capabilities, which are particularly important for mid-to-long-range interactions.

Research Outcomes

• Key Results:

  1. Two user studies validated the technology and user experience of TriPad:
     • Experiment 1: Verified the technical feasibility of gesture recognition, achieving over 95% accuracy across all surfaces and event types.
     • Experiment 2: Compared the performance of indirect and direct input modes, finding that TriPad reduces fatigue and improves precision, especially on horizontal surfaces.
  2. Significantly reduced the likelihood of false touch activation, while enabling smooth operation across multiple surfaces (glass, wood, drywall).

• Advantages and Comparisons:

  • Compared to direct input (e.g., mid-air touch or ray-casting selection), the indirect input mode significantly reduces user fatigue.
  • Horizontal TriPad outperformed vertical TriPad in a range of interaction tasks and surpassed traditional direct input methods in certain scenarios.
  • Achieved high precision across multiple surfaces without relying on external sensors, demonstrating "broad applicability."

• Experiment or Evaluation Details:

  1. Experiment 1 Validation:
     • Low false recognition rate, with only one false touch (Phase 1).
     • Actual touch plane recognition deviation was approximately 3.5°, compensated by combining hand position and virtual plane data.
  2. Experiment 2 Comparative Analysis:
     • Pointing Task: In precise target selection, the indirect input mode achieved a better speed-accuracy tradeoff.
     • Pursuit Task: Indirect input performed better in tracking distant targets but was slightly inferior to direct touch in close-range continuous dragging tasks.
  3. Survey results showed that users rated horizontal TriPad highest for comfort and low fatigue in subjective experience.

• Limitations and Future Directions:

  • Limitations:
    • Support for plane orientations is mainly limited to horizontal and vertical; non-standard orientations or complex geometric surfaces were not explored.
    • The design of a fixed control-display gain (CD-gain) may not be fully optimized.
    • The strategy does not account for industrial recommendations on target size intensity or simulations of broader contexts.
  • Future Work:
    1. Extend support for two-handed operations, such as defining larger planes or gestures on curved surfaces.
    2. Enhance expressiveness by utilizing "idle fingers" and expand functionality for more complex interfaces.
    3. Optimize control-display gain and gesture interpretation strategies through more in-depth experiments.