HybridTrak: Adding Full-Body Tracking to VR Using an Off-the-Shelf Webcam

Title of the Paper

HybridTrak: Adding Full-Body Tracking to VR Using an Off-the-Shelf Webcam

Paper Information

• Domain: Virtual reality, full-body tracking, computer vision
• Keywords: full-body tracking, virtual reality, computer vision, hybrid tracking, deep learning, RGB camera, pose estimation, SteamVR, user study, machine learning

Research Background and Problem

• Identified Problem: Current virtual reality devices primarily focus on head and hand tracking, lacking effective support for full-body tracking. Traditional full-body tracking systems require complex setups (e.g., external cameras and markers), while existing upper-body tracking using wearable cameras performs poorly for lower-body tracking.
• Significance: Full-body tracking enhances the sense of presence in virtual reality, supports interaction through body posture, and improves social expression among users.
• Motivation and Related Work:
  • Existing full-body tracking solutions either rely on bulky hardware or expensive RGBD cameras, often requiring additional calibration.
  • Studies have shown that upper-body tracking based on head-mounted cameras (inside-out tracking) suffers from occlusion issues, resulting in poor lower-body tracking performance.
  • This paper proposes a hybrid system combining uncalibrated external RGB cameras and built-in upper-body tracking to address the shortcomings of current technologies.

Solution

• Method: A new system named HybridTrak is introduced, which integrates 2D full-body pose data from uncalibrated RGB cameras with 3D upper-body position data from built-in VR systems. A deep learning neural network generates consistent 3D full-body tracking results.
• Innovations:
  1. Achieves high-precision full-body tracking using a single RGB camera and existing upper-body tracking systems.
  2. Introduces a fully neural network to fuse 2D camera data and 3D upper-body data, generating stable 3D poses.
  3. Directly produces waist and foot tracking data compatible with SteamVR.
• Implementation Steps:
  1. Data Input: Extract 2D poses from the RGB camera and 3D upper-body position data from the head-mounted display.
  2. Deep Learning Transformation: Use a pose transformation neural network to map 2D and 3D inputs into VR's 3D coordinate space.
  3. Virtual Device Simulation: Simulate virtual tracking points for the waist and feet using SteamVR drivers, enabling plug-and-play compatibility with VR applications supporting full-body tracking.
  4. Training Process: Train the neural network on Human3.6m and MPI-INF-3DHP datasets, utilizing synthetic data to enhance model performance.

Research Results

• Specific Outcomes:
  1. On the Human3.6m and MPI-INF-3DHP datasets, HybridTrak outperformed baseline methods using RGBD cameras in joint position error (MPJPE) and angular error (MPJRE).
     • HybridTrak's MPJPE was 0.098 meters, compared to the RGBD camera baseline's 0.136 meters.
     • HybridTrak's MPJRE was 0.282 radians, while the RGBD camera baseline's was 0.609 radians.
  2. User studies demonstrated that HybridTrak produced more accurate and natural poses compared to KinectToVR and upper-body tracking alone.
     • In experiments, users achieved a 99% pose recognition rate under HybridTrak conditions, significantly higher than other methods.
     • On a 7-point Likert scale, users rated pose naturalness and transition smoothness significantly higher for HybridTrak.
  3. The proposed fully neural network method surpassed the HybridTrak-transform method (based on transformation matrices) in computational efficiency and robustness.
• System Advantages:
  • Compared to most existing RGB or RGBD tracking algorithms, HybridTrak delivers higher accuracy with less hardware and lower resource requirements.
  • The system is calibration-free and plug-and-play, making it particularly suitable for general consumers.
  • Effectively addresses body occlusion issues, especially the disparity in occlusion between upper and lower body in virtual environments.
• Experimental Results:
  • Quantitative: On the MPI-INF-3DHP dataset, HybridTrak's MPJPE significantly outperformed the VNect algorithm (0.138 meters vs. 0.455 meters).
  • Qualitative: Users consistently rated HybridTrak-generated poses as natural and easy to recognize.
• Limitations and Future Directions:
  1. Limitations:
     • The current system requires a dedicated GPU for 2D pose estimation.
     • The training datasets used have limited diversity in skeleton sizes, which may not be user-friendly for individuals with extreme body types.
     • Certain complex poses (e.g., crossed-leg movements) remain unresolved in the output results.
  2. Future Directions:
     • Expand training dataset diversity through crowdsourcing or synthetic data generation.
     • Integrate depth camera information or optimize 2D pose detection to enable operation on low-spec hardware.
     • Develop specialized HybridTrak models for specific VR applications to improve accuracy and real-time performance in targeted scenarios.