T2IRay: Design of Thumb-to-Index based Indirect Pointing for Continuous and Robust AR/VR Input

Research Background and Issues

• Identified Problems or Challenges:

  • Existing micro-gesture interactions primarily focus on discrete inputs, making it difficult to support continuous interaction modes.
  • Current AR/VR systems often rely on freehand pointing inputs, which require the entire hand to remain within the camera's field of view. This leads to physical fatigue and limited interaction flexibility.
  • In micro-gesture interactions, the degrees of freedom provided by existing methods (e.g., 1-DOF or simple trajectory drawing) are insufficient for achieving truly continuous pointing operations.

• Significance:

  • The AR/VR domain requires natural and seamless interaction methods, and micro-gesture interactions based on thumb-to-index finger (T2I) have the potential to become an efficient solution.
  • Improving existing micro-gesture interactions can significantly enhance user experience, reduce fatigue, and increase input accuracy and flexibility.

• Research Motivation and Related Work:

  • Micro-gestures based on thumb-to-index finger interactions have already demonstrated potential in several fields, such as light touch or sliding gestures for input. However, most of these studies are limited to discrete interactions.
  • Indirect input methods have advantages in reducing hand fatigue and improving fine motor control, but current approaches still rely on fixed interaction planes, lacking flexibility.
  • Combining head movements with dynamic thumb-to-index finger gestures has the potential to explore high degrees of freedom in interaction, motivating the authors to design the T2IRay system.

Solution

• Proposed Method or Solution:

  • A novel indirect pointing technique—T2IRay—is proposed, utilizing the relationship between the thumb and index finger as a local coordinate system to enable continuous input based on micro-gestures, while integrating head movements to expand the interaction space.
  • T2IRay includes continuous tracking and selection mechanisms, designed with the following goals:
    1. Provide precise and smooth motion control.
    2. Offer familiar input metaphors for quick user adoption.
    3. Support high controllability without requiring direct visual attention.

• Innovations:

  1. Extending T2I micro-gestures to fully continuous input and proposing a mapping method based on local coordinates.
  2. Integrating head orientation and micro-gestures to enable context-aware pointing control.
  3. Systematically studying the 3D interaction characteristics of micro-gestures in AR/VR, including depth, amplitude, and mapping methods.

• Implementation Steps and Techniques:

  1. Local Coordinate System: Construct a local coordinate plane using key joints of the index finger and thumb, defining the thumb's motion range.
  2. Pointing Mapping:
     • Employ four mapping methods (linear, logistic, quadratic, interpolation) and validate their effectiveness through Fitts’ Law studies.
     • Map thumb positions to a virtual plane within a 2-meter range for cursor control.
  3. Head Involvement: Combine head orientation with finger control to achieve more natural 3D input.
  4. Selection Mechanism (Thumb-Click) Design:
     • Explore the dynamic characteristics of clicking in two behavioral modes (thumb touching the index finger surface and hovering).

Research Outcomes

• Specific Results:

  • T2IRay is the first system to achieve continuous input based on T2I micro-gestures, validated through rigorous user studies.
  • Pointing Evaluation: Logistic mapping outperformed other methods in terms of speed and accuracy.
  • Click Performance Analysis:
    • Thumb-click behavior exhibited greater stability, more precise control, and more consistent speed characteristics in the "touch-skin" mode.
    • Variations in force and clicking behavior depending on direction and position suggest the need for region sensitivity settings to optimize system responsiveness.

• Advantages Compared to Existing Solutions:

  • Compared to direct input techniques (e.g., hand ray cursors), T2IRay performs better in reducing arm fatigue and achieving high-resolution input.
  • Provides a flexible 3D interaction method, overcoming the limitations of traditional discrete micro-gestures.

• Experimental or Evaluation Results:

  1. Experiments based on Fitts’ Law validated system performance across different mapping methods and distances:
     • Logistic mapping achieved a better balance between speed and accuracy.
     • Depth variations had minimal impact on total completion time but affected gesture precision to some extent.
  2. T2IRay’s motion demonstrated unique challenges in precision for upward, downward, and diagonal directions, further emphasizing the need for dynamic adjustments.
  3. Pointing experiments revealed significant differences between the two clicking behavior modes, providing data to optimize interaction parameters.

• Limitations and Future Directions:

  1. Limitations:
     • Standardized comparisons with other traditional methods have not yet been achieved.
     • User studies were primarily conducted in seated positions, without sufficient consideration of dynamic scenarios or diverse user groups (e.g., variations in hand size and flexibility).
     • The current click-based selection method may be constrained by hardware precision.
  2. Future Directions:
     • Broad exploration of head and gesture integration to improve multimodal interaction efficiency.
     • Development of depth adaptability and real-time stabilization mechanisms (e.g., anti-shake algorithms).
     • Expansion to more complex scenarios, such as dynamic walking environments and multitasking contexts.

Conclusion

T2IRay demonstrates an innovative indirect pointing input technique based on T2I micro-gestures, providing a continuous, stable, and user-friendly solution for AR/VR interaction. Through systematic experiments, this study validates the feasibility and potential of T2IRay and offers valuable design insights. However, further exploration of comparisons with other input methods and adaptability to diverse application scenarios is necessary to achieve truly widespread adoption.