ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Title of the Paper

ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Paper Information

• Research Area: Human-Computer Interaction (HCI), Virtual Reality (VR), Body Motion Capture
• Keywords: Virtual Reality, Pose Tracking, Motion Capture, Controller Cameras, Full-Body Pose Tracking, Computer Vision, Deep Learning, User Experience, Camera Fusion, IMU

Research Background and Problem Statement

• Problems or Challenges:

  1. Most current VR systems fail to capture lower body poses, resulting in users seeing only a hollow or simplified representation (e.g., shadows) of themselves in virtual environments, which reduces immersion and embodiment.
  2. Existing solutions relying on external hardware (e.g., wearable sensors or environment-mounted cameras) perform well but significantly reduce user adoption due to additional hardware costs and setup complexity.
  3. Using cameras on head-mounted devices (HMDs) to address this issue faces challenges such as limited field of view, occlusion, and self-occlusion (e.g., by hands or clothing). These limitations may worsen as HMDs become lighter and thinner.

• Significance:

  1. Improving the realism of VR avatars (including the lower body) enhances the immersive and interactive experience in virtual environments.
  2. Developing practical and cost-effective methods that integrate seamlessly into existing consumer-grade VR hardware can drive broader user adoption and expand application scenarios.

• Research Motivation and Related Work: The authors point out that existing solutions (e.g., external sensors, HMD cameras, wearable devices) have their pros and cons, but there is no simplified, integrated approach to achieve full-body capture using VR controllers. They also observe that during most VR interactions, users' hands are typically positioned in front of their bodies, making them suitable for capturing body images. This observation motivated the development of a controller-based camera system.

Solution

• Method or Solution: The authors propose a novel solution that integrates two fisheye cameras into VR hand controllers, leveraging their wireless communication, computational capabilities, and power systems. The method includes:

  1. Using the cameras on the VR controllers to capture body poses, as they can view the user's body for most of the VR interaction time.
  2. Combining multi-camera views, 3D pose estimation, and inverse kinematics-based human body modeling.

• Innovations:

  1. The solution does not require additional sensors or external hardware, relying solely on integrating cameras into existing VR controllers.
  2. Fisheye cameras provide a wide field of view for capturing the user's body, improving robustness against occlusion.
  3. The authors propose a multimodal neural network combining 2D multi-view pose estimation and VR sensor data to estimate limb poses and support various VR actions.

• Implementation Steps and Key Techniques:

  1. Hardware System: Install two fisheye cameras on Oculus Quest 2 controllers and set up wireless data transmission.
  2. Data Processing Pipeline: Capture video input, synthesize images, correct fisheye distortion, and project and stitch them into panoramic views.
  3. Pose Estimation:
     • Use OpenPose to extract 2D keypoint data.
     • Combine 2D data with IMU data from the headset and controllers, input them into a multi-input neural network, and predict the 3D poses of 17 joints.
  4. User Interaction and Evaluation: Use inverse kinematics to animate avatars based on the final pose output and evaluate the system's performance through experiments.

Research Findings

• Specific Results:

  1. The proposed system achieved full-body pose capture with an average Euclidean error of 6.98 cm for 3D joint positions.
  2. The method performed well in capturing typical VR actions such as walking, kicking, and squatting, demonstrating its practicality.

• Advantages Compared to Existing Solutions:

  1. Compared to HMD-based camera systems, the controller-mounted cameras are less affected by body occlusion, especially for the lower body.
  2. No additional hardware is required, as the system fully utilizes existing VR controllers, making it cost-effective and easy to set up.
  3. The input data combines camera views and IMU data, further enhancing accuracy and robustness.

• Experimental or Evaluation Results:

  1. User experiments showed that the system could accurately capture movements in various postures, with significant improvements in lower body pose tracking.
  2. In challenging scenarios (e.g., clothing occlusion, low lighting), the system occasionally made errors but still achieved significantly lower errors (6.98 cm) compared to common 3D human pose estimation models like CenterHMR (24.6 cm).
  3. The dynamic performance requires optimization, as the current pipeline processes at 7.2 FPS with a total latency of approximately 297 milliseconds.

• Limitations and Future Directions:

  1. Occlusion Issues: Pose inference fails when arms are placed at the sides of the body or when clothing occludes the body. Adding more cameras (e.g., on the bottom of the controllers) could address this.
  2. Environmental Constraints: Strong lighting, shadows, and complex backgrounds impact accuracy, requiring future improvements.
  3. Performance Optimization: The system is currently a proof-of-concept and needs optimization for processing speed and real-time performance to meet commercial VR system requirements.
  4. Extended Functionality: Future work could integrate HMD cameras to enhance upper body tracking or combine data from other wearable devices like smartwatches to further improve robustness.

Conclusion

By integrating cameras into VR controllers, the authors provide a novel solution for lower body motion capture, enhancing immersion while reducing costs and complexity. Although current performance limitations exist, the technology has broad potential applications. Future optimizations and extensions could make this approach a valuable tool for VR gaming, social interactions, and health applications.