Occlusion and hand pose cause tracking loss
Aliases: self-occlusion · palms inward · visual dropout
What it is
Palms toward the chest, fingers curled into a fist, or one hand reaching from behind the other — and the outward-facing cameras on the headset brim suddenly have nothing to read. Bare-hand tracking is lost mainly to hand tracking occlusion: the cameras need skin landmarks, and those landmarks vanish behind one’s own palm, the other hand, the torso, or a real object.
This is not a network drop, and not merely “the hand left the tracked volume.” The hand is still in front of the body, still doing the intended motion. From the cameras’ point of view it has simply ceased to be a hand.
Why it happens
Headset hand tracking is inside-out vision: sensors near the forehead looking out and down. Observable poses are therefore clipped by a small cone. Palms toward the face show the backs of the fingers; palms toward the body with fingertips inward show a flesh-coloured silhouette, self-occluded knuckles, and mushy depth. Crossed hands, one hand around the other wrist, a hand against the torso — each adds a second occluder.
When landmarks disappear the estimator still emits a pose, filling gaps with kinematic priors: the wrist is still there, the fingers continue last frame’s flexion. For a short time this looks like “still tracking.” If the facing does not recover, the guess slides toward a legal but untrue shape — fingers through the palm, a hand kinked at the elbow. What the user sees is a broken hand; what happened is that the cameras stopped seeing it some time ago.
Studying it
Run robustness tests on poses known to hide the cameras, not only on an open palm facing forward.
Independent variables: palm facing (toward camera / toward torso / sideways), overlap of the two hands, distance from hand to torso, lighting and background texture. Dependent variables: number of visible landmarks, duration of loss, joint-angle error against a mocap ground truth, the moment users report “the hand is gone.”
A mocap glove or external multi-camera rig can supply truth. The headset’s own confidence score is optimistic: facing is already wrong while the score is still high. Defining loss as “confidence below a threshold” versus “joint error past N degrees” produces entirely different incidence rates.
Where it stops holding
Outside-in lighthouses and inertial gloves do not look at the hand with headset cameras; their occlusion pattern is different and this cause does not dominate. Extra cameras on the chest or shoulders widen the cone and ease facing failures. Dark skin, dark cuffs, and strong backlight bring the “visible” threshold forward so landmarks vanish from exposure before occlusion happens. One-handed tasks that keep the palm toward the headset (an open menu) rarely trip this; tasks that fish objects from the sides of the body, pinch both hands together, or rest a hand behind the hip trip it constantly. Children’s hands pack landmarks into a small blob; a slight facing change deletes the whole hand from the model.
Applying it
- Put finger-precise actions inside the cone the headset cameras can see: in front of the body, palm slightly up or out. Do not design “palm on the belly” or “fish a virtual pocket at the hip.”
- Do not leave two-handed operations in a crossed or fully covering pose for long.
- When the estimator starts filling with priors, give a perceptible downgrade (translucent hand, softened outline). Do not wait until fingers have folded into the palm.
- How to check: have people perform the product’s real actions and watch the headset camera debug view, not a third-person preview. Any stretch where the debug view turns the hand into a colour blob or deletes it is a facing or occlusion the interaction failed to avoid.