HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction Using Inertial Sensing

Paper Title

HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensing

Paper Information

• Field of Study: Virtual Reality (VR) and Augmented Reality (AR) user interaction technologies
• Keywords: Virtual reality, hand tracking, inertial sensing, inertial tracking, proprioceptive interaction, out-of-view interaction, sensor fusion

Research Background and Problem

• Identified Problems or Challenges: Current virtual reality devices primarily rely on visual tracking to monitor hand or controller movements, limiting user interaction in areas outside the field of view. Existing solutions often require additional camera hardware, which can increase computational load, power consumption, and the weight of head-mounted displays.
• Significance: Humans possess spatial awareness beyond the 210-degree field of view, allowing them to interact with objects outside their visual range through proprioception. This capability offers the potential for more natural interactions. Expanding the interaction range can enhance user efficiency while reducing physical strain from frequent head movements.
• Research Motivation and Related Work: This paper explores how to extend hand tracking beyond the field of view using simple hardware (a single inertial sensor) and a data-driven approach to accurately detect hand position and gestures outside the tracking range. Related methods include IMU-based body or limb tracking and studies on human spatial memory and proprioceptive abilities.

Solution

• Proposed Method or Solution:

  • HOOV Method: An inertial tracking technique based on a wrist-worn device using a 6-axis IMU (comprising a 3-axis accelerometer and a 3-axis gyroscope), combined with head-mounted display visual tracking data to predict hand position and posture (6D).
  • Data-Driven Architecture: Deep learning methods, including Transformer and RNN modules, are employed to process time-series data for position and orientation estimation.
  • Gesture Detection: Pinch actions are detected through changes in acceleration signals, and haptic feedback is integrated to enhance the interaction experience.

• Innovations:

  1. A low-power solution entirely based on inertial sensors to extend the interaction range of AR/VR devices.
  2. A time-series learning architecture for short-term motion prediction, reducing inertial drift errors.
  3. Integration of haptic feedback devices to provide immediate user feedback.

• Implementation Steps and Key Techniques:

  1. Data Collection: IMU signals and head position/orientation data are collected via the wrist device and head-mounted display.
  2. Data Prediction: An extended Kalman filter is used for initial orientation estimation, followed by deep learning models processing time-series data to predict 3D positions.
  3. Input Event Recognition: Pinch gestures are identified through changes in acceleration signals, with vibration motors providing feedback.

Research Results

• Specific Results:

  • HOOV achieved continuous estimation of 3D hand positions outside the field of view, with a mean absolute error (MAE) of 7.77 cm (compared to a high-precision optical motion tracking system).
  • In real-time interactions, the success rate for correctly detecting grasp and release actions exceeded 86%.

• Advantages:

  • Compared to existing IMU-based wrist tracking methods, HOOV significantly reduced short-term motion prediction errors (approximately 7 cm), outperforming many real-time or offline methods.
  • Users could interact with 17 targets in the scene without turning their heads, improving interaction speed (task completion time reduced by approximately 15-19%).
  • In terms of user experience, participants preferred the non-visual interaction condition enabled by HOOV.

• Experiment or Evaluation Results:

  • The paper validated HOOV's accuracy and utility through two user studies (total of 22 participants):
    1. Offline Evaluation: Using the OptiTrack optical system, HOOV simulated out-of-view tracking tools with an average success rate of approximately 85%.
    2. Real-Time Interaction Evaluation: HOOV's predictions fully drove user task execution, maintaining a success rate of around 86%.
  • Gesture Detection Accuracy: Recall rate of 94.85% and precision of 98.05%.

• Limitations and Future Directions:

  1. Training Data Limitations: Larger datasets are needed to improve personalized adaptation, especially for adjustments based on different user body types.
  2. Inertial Drift Issues: Hand position errors increase over interaction time; future work could integrate other localization technologies (e.g., optical tracking) to further reduce drift.
  3. Device Dependency: The current system requires users to wear a dedicated wrist device; future research could explore integrating HOOV into existing smartwatches or devices.
  4. Extension to Mobile Scenarios: The current study is limited to static scenarios; future work could explore robustness and applications in dynamic environments.
  5. Computational Complexity: Model power consumption and real-time efficiency need further optimization to support deployment on lower-performance platforms.

Conclusion

HOOV opens new possibilities for AR/VR interaction technologies by significantly enhancing user convenience and speed through extended tracking ranges. Its low-cost solution based on inertial sensors has broad application potential in areas such as gaming, sports training, and medical rehabilitation. Future work could focus on improving tracking accuracy, expanding scenario types, and deeply integrating with ecosystem hardware systems to achieve more natural out-of-view interaction experiences.