HAI-AR: Exploring Hand-Anchored Interfaces in Augmented Reality while Walking

Research Background and Issues

• Identified Problems or Challenges: Existing augmented reality (AR) user interfaces (UIs) face multiple limitations when users are walking. For example, they are often fixed in a specific position within the field of view, obstructing the forward view, and require manual adjustment of the UI position after user movement. This results in suboptimal interaction experiences in dynamic scenarios.

• Significance of the Research: With the increasing adoption of AR head-mounted devices, researching how to provide stable and efficient UI experiences in dynamic environments is critical for enhancing the practicality and universality of AR systems, especially in scenarios where users need to interact with digital information while walking.

• Motivation and Related Work: Although prior research has explored various reference frameworks (e.g., head-referenced, torso-referenced), these solutions often encounter issues related to stability, usability, or visual interference during user movement (e.g., walking). The authors propose using hand-referenced frameworks (e.g., palm, forearm, pinch gestures) for UIs, which potentially reduce visual interference and provide a more natural access method. However, the performance of hand-referenced frameworks in walking scenarios remains underexplored.

Solution

• Proposed Method or Solution: The authors propose a design method for "Hand-Anchored Interfaces" and focus on testing six specific hand-referenced frameworks, particularly the innovative "Pinch Grip With Offset" (PGWO) approach. The study aims to improve user interaction efficiency, accuracy, and comfort through offset distances and anchoring points.

• Innovations:

  1. Combining the naturalness of hand movements (e.g., pinch gestures) with offset distances to create intuitive and ergonomic UI designs.
  2. Introducing a novel method that integrates pinch gestures with offset adjustments, providing a coordinated hand-eye-UI interaction experience.
  3. Conducting the first systematic comparison of the performance of various hand-referenced frameworks under both static and dynamic conditions.

• Implementation Steps and Key Technologies:

  • Experimental Design:
    1. Study 1: Evaluate the speed, accuracy, and user preferences of six hand-referenced frameworks (Forearm No Offset, Forearm With Offset, Palm No Offset, Palm With Offset, Pinch Grip No Offset, Pinch Grip With Offset) through a Fitts' Law-based target selection task.
    2. Study 2: Compare "Pinch Grip With Offset" (PGWO) with head-referenced and torso-referenced frameworks to assess their advantages in target selection and information access tasks.
  • Key Technologies:
    1. Utilizing hand-tracking technology from Meta Quest Pro to recognize user hand positions and movements.
    2. Implementing dynamic UI behavior in AR environments using the Unity development environment and Meta XR SDK.
    3. Analyzing data based on Fitts' Law, task completion time, error rates, and NASA-TLX workload scales.

Research Outcomes

• Specific Findings:

  1. PGWO outperformed all six hand-referenced frameworks, offering lower movement times, higher selection accuracy, and reduced user workload.
  2. In further comparisons with head-referenced and torso-referenced frameworks, PGWO demonstrated superior performance in target selection tasks (speed and accuracy) and information access tasks (task completion time, error rates, etc.).
  3. Users highly preferred the PGWO framework, citing its intuitive interaction, better sense of control, and reduced physical effort.

• Advantages Over Existing Solutions:

  1. Stability and Intuitiveness: PGWO effectively reduces UI instability caused by movement in torso or head-referenced frameworks.
  2. Reduced Visual Interference: Compared to head-referenced frameworks, PGWO avoids obstructing the forward field of view with fixed UIs.
  3. Ergonomic Benefits: PGWO allows users to leverage natural hand movements and offset distances, making interactions more comfortable and less fatiguing.

• Experimental and Evaluation Results:

  1. In the first phase of the experiment, PGWO achieved a task completion time of 1.36 seconds, significantly faster than other reference frameworks.
  2. In comparisons with head-referenced and torso-referenced frameworks, PGWO not only achieved faster task times (e.g., 3.95 seconds for information access) but also significantly reduced error rates.
  3. Fitts' Law regression analysis indicated that PGWO exhibited the strongest adaptability to task difficulty (R² = 0.95).

• Limitations and Future Directions:

  1. Testing was conducted only with right-handed users, and interactions with the left hand or non-dominant hand may yield different results.
  2. The offset distance was fixed; future studies could explore the impact of dynamic or customizable offset values.
  3. Experimental conditions were idealized (e.g., indoor environments); future research should extend to real-world scenarios (e.g., outdoor or crowded environments).
  4. The current study focused on AR environments; future work could investigate performance in fully immersive virtual reality (VR) settings.

Conclusion

Through experimental evaluation of six hand-referenced frameworks, the authors proposed the Pinch Grip With Offset (PGWO) hand-referenced framework design method, which demonstrated significant advantages in dynamic augmented reality scenarios. Whether for target selection or information retrieval tasks, this method exhibited efficient and intuitive human-computer interaction performance, providing feasible design recommendations for future dynamic AR interface development.