DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

Paper Title

DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

Publication Info

• Topic area: Egocentric hand pose estimation using dorsal skin deformation.
• Keywords: Egocentric hand tracking, dorsal features, self-occlusion, 3D hand pose estimation, dense visual featurization, XR interactions, skin deformation, gesture recognition, isometric click detection, computer vision.

Background and Problem

• Problem / challenge: Egocentric hand pose estimation suffers from frequent self-occlusion, where fingers are partially or fully obscured, limiting the performance of state-of-the-art (SOTA) models that rely on hand silhouettes or geometric representations.
• Significance: Accurate hand pose estimation is critical for enabling intuitive gesture-based control in XR systems and other interactive technologies. Addressing self-occlusion can unlock more robust and seamless interactions.
• Motivation and related work: Prior works have focused on geometric features, multi-view setups, or wearable sensors, but these approaches are limited by occlusion, infrastructure requirements, or restricted user mobility. Dorsal skin deformation, which remains visible during occlusion, has been underutilized in egocentric hand tracking.

Solution

• Proposed approach: DeltaDorsal, an end-to-end system that uses dorsal skin deformation features for egocentric hand pose estimation, leveraging dense visual featurizers and a dual-stream delta encoder.
• Novelty:
  1. Introduction of a dorsal feature-based approach to mitigate self-occlusion in egocentric hand tracking.
  2. Development of a dual-stream architecture that contrasts dorsal features from a relaxed reference pose and a dynamic gesture.
  3. Collection of a high-resolution dataset of 172,222 frames capturing dorsal hand features across 17 gestures from 12 participants.
  4. Demonstration of new interaction paradigms, such as isometric click detection, using dorsal skin deformation.
• Procedure and key techniques:
  1. Use of a reference image of the hand in a neutral pose for alignment and comparison with gesture images.
  2. Dense feature extraction using a pretrained DINOv3 Vision Transformer.
  3. Feature delta and cosine similarity computation to isolate dorsal deformations.
  4. Prediction of 3D hand pose parameters using a regression head and MANO model.

Results

• Concrete findings:
  • DeltaDorsal achieved a mean per joint angular error (MPJAE) of 6.41°, outperforming HaMeR (6.74°) and HandOccNet (11.27°).
  • Performance was less affected by self-occlusion (m = −4.59°) compared to HaMeR (m = −8.42°) and HandOccNet (m = −9.11°).
  • Skin tone had negligible impact on performance (η² = 0.0055).
  • Enabled accurate detection of gestures like taps (RMSE: 26.23° for index finger) and pinches (RMSE: 16.81 mm for index finger).
  • Demonstrated isometric click detection accuracy of 0.85 for in-air gestures and 0.84 for surface gestures.
• Advantage over baselines:
  • Outperformed HaMeR and HandOccNet in occluded scenarios, particularly for thumb and index finger movements.
  • Achieved comparable or better performance with a smaller model size (300M parameters vs. HaMeR’s 632M).
  • Demonstrated robustness across different skin tones and occlusion levels.
• Experiments / evaluation:
  • Leave-one-subject-out (LOSO) cross-validation on a high-resolution dataset.
  • Comparison with HaMeR and HandOccNet using metrics like MPJAE and PA-MPJPE.
  • Ablation studies on image size and model backbone to assess performance trade-offs.
  • Application studies on tap, pinch, and isometric click detection.
• Limitations and future work:
  • Limited generalizability to in-the-wild scenarios due to constrained dataset and controlled conditions.
  • Dependency on high-resolution camera feeds for optimal performance.
  • Future work includes expanding datasets to diverse populations, integrating temporal features, and optimizing for lightweight hardware.

Summary

DeltaDorsal introduces a novel approach to egocentric hand pose estimation by leveraging dorsal skin deformation features, addressing the challenge of self-occlusion. The system outperforms SOTA baselines like HaMeR and HandOccNet, achieving an MPJAE of 6.41° and demonstrating robustness across occlusion levels and skin tones. It enables new interaction paradigms, such as isometric click detection, and shows promise for deployment on lightweight devices. While further work is needed to generalize to in-the-wild scenarios, DeltaDorsal represents a significant step toward more robust and flexible hand tracking in XR systems.