WristPP: A Wrist-Worn System for Hand Pose and Pressure Estimation

Paper Title

WristPP: A Wrist-Worn System for Hand Pose and Pressure Estimation

Publication Info

• Topic area: Wrist-mounted systems for hand pose and pressure estimation in human-computer interaction.
• Keywords: wrist-mounted sensing, hand pose estimation, pressure estimation, Vision Transformer, extended reality, touchpad interaction, mid-air gestures, human-computer interaction, wearable devices, fisheye camera.

Background and Problem

• Problem / challenge: Simultaneous estimation of 3D hand pose and pressure in mobile scenarios is challenging due to articulatory complexity, occlusion, and the lack of portable, low-cost solutions.
• Significance: Accurate hand pose and pressure sensing is critical for immersive and efficient human-computer interaction, enabling applications such as XR, virtual touchpads, and pressure-sensitive input.
• Motivation and related work: Prior systems like environment-mounted cameras, head-mounted displays, and glove-based wearables have limitations in mobility, occlusion handling, comfort, and cost. Wrist-mounted systems show promise but lack pressure estimation capabilities.

Solution

• Proposed approach: WristP2, a wrist-mounted system with a fisheye RGB camera, reconstructs 3D hand pose and per-vertex pressure in real time using a Vision Transformer-based architecture.
• Novelty:
  1. Introduction of an eyes-free, low-cost wrist-mounted device for joint pose and pressure estimation.
  2. Development of a large-scale dataset with aligned hand meshes and surface pressure annotations.
  3. Design of an extrinsics-aware Hand–VQ–VAE architecture for joint pose and pressure prediction.
  4. Demonstration of touchpad-level efficiency and robust pressure control in user studies and real-world applications.
• Procedure and key techniques:
  • Use of a fisheye RGB camera mounted on the wrist for proximal sensing.
  • Vision Transformer backbone with joint-aligned tokens for pose and pressure estimation.
  • Tokenization of hand meshes via Hand–VQ–VAE for discrete latent space representation.
  • Multi-task learning with cross-attention mechanisms for pose reconstruction and pressure regression.

Results

• Concrete findings:
  • Pose estimation achieved MPJPE of 2.9 mm and MJAE of 3.2°.
  • Pressure estimation metrics included Contact IoU of 0.712, Vol.IoU of 0.618, and foreground pressure MAE of 10.4 g.
  • Wrist-camera extrinsics estimation yielded rotation error of 2.3° and translation error of 8.9 mm.
• Advantage over baselines:
  • WristP2 outperformed head-mounted and environment-mounted baselines in pose and pressure accuracy, achieving 5–15× lower pose errors and 95.2% F1-score for contact prediction.
  • In a large-display Whac-A-Mole task, WristP2 reduced reaction time by 35–40% and error rate by one-third compared to head-mounted methods.
• Experiments / evaluation:
  • Offline evaluation across varying illumination and surfaces showed robustness.
  • User studies demonstrated touchpad-level pointing efficiency, multi-finger pressure control, and virtual touchpad functionality.
  • Application study in a large-display task highlighted ergonomic and performance advantages over head-mounted systems.
• Limitations and future work:
  • Challenges in hand-object interaction, visual obstruction, and bimanual occlusion.
  • Need for slimmer hardware form factors and real-time on-device deployment.
  • Plans to extend datasets for object-centric interactions and explore multimodal sensing.

Summary

WristP2 is a wrist-mounted system that combines pose and pressure estimation using a fisheye RGB camera and Vision Transformer-based architecture. It achieves high accuracy in pose reconstruction (MPJPE: 2.9 mm) and pressure estimation (Contact IoU: 0.712), outperforming head-mounted and environment-mounted baselines. User studies validate its effectiveness for mid-air gestures, multi-finger pressure control, and virtual touchpad tasks, while a real-world large-display application demonstrates ergonomic and performance benefits. Future work aims to address hand-object interaction, occlusion challenges, and hardware optimization for broader adoption.