Can We Infer Object Pose Changes from Hand Movements?

Paper Title

Can We Infer Object Pose Changes from Hand Movements?

Publication Info

• Topic area: Object tracking in Augmented Reality using hand dynamics.
• Keywords: Object tracking, hand dynamics, Augmented Reality, pose estimation, hand-object interaction, grasp detection, release detection, computer vision, HOT3D dataset, hybrid tracking.

Background and Problem

• Problem / challenge: Robust tracking of manipulated objects in AR is difficult due to occlusions caused by hands, appearance changes, and shape variations of objects. Existing computer vision methods often fail to maintain continuous tracking during manipulation.
• Significance: Continuous object tracking is critical for seamless AR experiences, such as augmented procedural instructions, object intelligence applications, and one-handed XR interfaces. Losing object pose disrupts user interaction and diminishes the AR experience.
• Motivation and related work: Previous work has focused on fiducial markers, 6DoF pose estimation, and computer vision-based methods, which struggle with occlusions and open-world conditions. Hand-centric approaches, like GripMarks, require user-dependent training and are limited to specific grips. This paper investigates whether object pose changes can be inferred from hand dynamics alone, filling a gap in lightweight, general-purpose tracking solutions.

Solution

• Proposed approach: Object-from-Hand (ObHa) – A hand-centric method that infers object pose changes from hand dynamics using built-in hand-tracking APIs.
• Novelty:
  1. Tracks object pose changes without requiring object-specific models or training.
  2. Robust to hand occlusion, object appearance changes, and shape variations.
  3. Supports generic grasp actions, enabling multiple ways of holding objects.
  4. Operates with low computational overhead, suitable for lightweight AR devices.
• Procedure and key techniques:
  1. Probe 1: Initial exploration of grasp detection using hand shape alone, identifying key issues like false positives and negatives.
  2. Probe 2: Refinement with spatial proximity and explicit release gestures, achieving 88% grasp detection accuracy and 97% release detection accuracy.
  3. Probe 3: Insights from the HOT3D dataset to refine heuristics for natural release detection, achieving 95% grasp detection accuracy and 88% release detection accuracy.

Results

• Concrete findings:
  • Probe 2: 88% grasp detection accuracy, 97% release detection accuracy.
  • Probe 3: 95% grasp detection accuracy, 88% release detection accuracy on the HOT3D dataset.
  • Temporal accuracy: Median grasp detection error of -0.099 s, release detection error of -0.297 s.
  • Spatial accuracy: Final placement error of ~5 cm in 16% of trials, with ~15° orientation offset in 5% of trials.
• Advantage over baselines:
  • Real-time performance (70 fps on Meta Quest 3).
  • No object-specific knowledge or training required.
  • High robustness to hand occlusion and object variability compared to HOI reconstruction and 6DoF trackers.
• Experiments / evaluation:
  • Probe 1: Formative study with 3 participants manipulating 9 objects.
  • Probe 2: Laboratory study with 12 participants, testing grasp and release detection on 5 objects.
  • Probe 3: Evaluation on 170 curated clips from the HOT3D dataset, analyzing natural object manipulations.
• Limitations and future work:
  • Challenges with two-handed interactions, multi-user scenarios, and non-prehensile actions.
  • Occasional unnatural release gestures required.
  • Dependence on hand-tracking system accuracy, especially under occlusion.
  • Future work includes extending to two-handed interactions, improving hand-tracking robustness, and integrating with computer vision pipelines.

Summary

The paper introduces Object-from-Hand (ObHa), a lightweight approach to object tracking in AR that infers pose changes from hand dynamics. Through three iterative probes, the method was refined to achieve robust grasp and release detection, with accuracies of up to 95% and 88%, respectively. ObHa operates without object-specific models, is robust to occlusion and object variability, and runs efficiently on lightweight AR devices. It can serve as a standalone tracker for rapid prototyping or complement computer vision in hybrid pipelines. Future work aims to address two-handed interactions, improve naturalness, and enhance tracking robustness.