3DRing: Enabling Low-Cost 3D Hand Position Tracking by Fusing Inertial and Low-Framerate Optical Sensing

Paper Title

3DRing: Enabling Low-Cost 3D Hand Position Tracking by Fusing Inertial and Low-Framerate Optical Sensing

Publication Info

• Topic area: Low-cost 3D hand tracking for AR/VR using optical-inertial fusion.
• Keywords: 3D hand tracking, low-framerate optical sensing, inertial measurement unit (IMU), optical-inertial fusion, reinforcement learning, extended Kalman filter, AR/VR interaction, computational cost reduction.

Background and Problem

• Problem / challenge: Existing hand-tracking systems rely heavily on high-framerate (HFR) optical sensors, which are computationally expensive and unsuitable for lightweight AR/VR devices. Low-framerate (LFR) optical sensing alone struggles with accuracy and latency, especially for dynamic hand movements.
• Significance: Reducing computational costs while maintaining tracking accuracy is critical for enabling hand tracking on lightweight and mobile AR/VR devices, which currently lack such capabilities.
• Motivation and related work: Prior work has explored LFR optical sensing and inertial sensing separately, but these approaches face challenges in balancing accuracy and latency. Optical-inertial fusion has been applied in other domains but has not been optimized for LFR hand tracking in dynamic 3D interactions.

Solution

• Proposed approach: 3DRing, a 3D hand position tracking framework that combines a single IMU ring with LFR optical data to achieve accurate, real-time tracking with reduced computational cost.
• Novelty:
  1. Introduction of a Deep Extended Kalman Filter (DEKF) that combines RNN-based velocity prediction with EKF for accurate motion prediction from sparse optical data and IMU input.
  2. Development of a reinforcement learning (RL)-based adaptive framerate strategy to dynamically select keyframes for calibration, optimizing the balance between accuracy and computational cost.
  3. Demonstration of significant computational cost reduction (to ~11% of conventional methods) while maintaining high tracking accuracy and interaction efficiency.
• Procedure and key techniques:
  1. Prediction Stage: Use an RNN to predict hand velocity from IMU data and integrate it with EKF to estimate hand position and rotation.
  2. Calibration Stage: Employ an RL model to adaptively select keyframes for optical calibration based on hand motion state.
  3. Evaluate the system using a 3D target selection task and compare it against baselines.

Results

• Concrete findings:
  • Achieved an average tracking error of 1.75 ± 0.18 cm using 6.61 FPS optical data.
  • Interaction efficiency reached 86.0% of the Meta Quest Pro (67 FPS).
  • Reduced computational cost to ~11% of conventional CV-based hand tracking methods.
• Advantage over baselines:
  • Reduced tracking error by 27.1% compared to LFR optical-only methods.
  • Outperformed vision-only and EKF-based baselines in accuracy and efficiency.
• Experiments / evaluation:
  • Offline evaluation: Cross-user validation on a dataset of 20 participants performing 3D target selection tasks.
  • Online evaluation: Real-time testing with 20 participants comparing 3DRing, Meta Quest Pro (67 FPS), and LFR Quest Pro (6.25 FPS).
  • Metrics: Tracking error, interaction efficiency, computational cost, and subjective user feedback.
• Limitations and future work:
  • Accuracy gap compared to HFR systems like Meta Quest Pro.
  • Fixed IMU ring orientation and reliance on index finger position as a proxy for hand position.
  • Hardware latency (~120 ms) due to BLE transmission and data processing.
  • Limited validation to basic 3D pointing tasks; future work should explore more complex interactions.

Summary

3DRing is a novel framework for low-cost 3D hand position tracking that fuses LFR optical data with inertial sensing from a single IMU ring. By integrating a DEKF for motion prediction and an RL-based adaptive framerate strategy, it achieves high tracking accuracy (1.75 cm error) and interaction efficiency (86% of Meta Quest Pro) while drastically reducing computational cost (~11% of conventional methods). Evaluations demonstrate its potential for lightweight AR/VR applications, though future work is needed to address accuracy, latency, and broader interaction scenarios.