Skeleton keypoints get worse as soon as something is hidden or far

What it is

A full-body skeleton crushes a person into joint coordinates. Those coordinates are not measured physical points. They are a model's estimate from images, and the estimate drifts as soon as occlusion and distance rise. An arm covering the torso, two people overlapping, furniture cutting the legs, standing so far that the body is a few dozen pixels: elbows and wrists jump, left and right legs swap, missing joints are filled from a prior. Near-field hand tracking looks at fingers. Full-body skeletons look at shoulders, hips, knees. The accuracy curves differ. Pinch numbers must not be copied onto full-body commands.

Why it happens

Monocular or depth pose estimation is visible pixels plus a body prior. Occlusion removes pixels, so the prior fills in “what a normal person should look like this frame”: a hidden hand sticks to the hip, or tunnels through the torso. Distance shrinks the person, joint spacing on the image falls to noise, and the symmetric legs are the first to swap. Temporal filters can smooth jumps and will also smooth real fast moves. Confidence usually falls with occlusion, but many applications still feed low-confidence joints into gesture recognition—hallucination as input. With several people, occlusion is mutual; accuracy drop is no longer something “better user stance” can fix alone.

Studying it

Report per-joint position error and left/right swap rate binned by occlusion type and distance, not only mean MPJPE. Occlusion must be real: self (hand over face), object (table over hip), person (two people crossing). Sweep distance radially. Drop low-confidence joints versus still feeding them, and watch gesture errors. Gold standard is multi-camera or optical markers; a single camera compared with itself treats the same prior's bias as stability.

Where it stops holding

Multi-camera rigs or inertial suits can ride through single-view occlusion, at deployment cost. A headset looking at its own hands is barely a full-body problem. If the prior is still a standing body, sitting and lying produce invented legs, and wheelchair users fail systematically. In the dark, infrared point clouds thin and the distance effect arrives early. Children and pets in frame get forced into an adult skeleton and sprout extra fake joints. Shadows that look like occlusion trigger the same prior fill.

Applying it

• Drive gestures only from high-confidence, unoccluded joints. On low confidence, freeze or degrade; do not let hallucination drive commands.
• Publish joint error binned by occlusion type and distance. Mean error from a laboratory frontal stand is not a full-body-channel test.
• Two people in frame is mandatory. “Please stand alone” is not a deployment assumption.

Related

• Same group: C4.13.2 Full-body moves are less socially acceptable than hand moves · C4.13.3 Body variation and prostheses mismatch skeleton models
• Adjacent: C4.10 Sensing volume and interaction distance · C4.31 Lighting, occlusion, and environmental conditions
• Search: pose estimation · keypoint occlusion · skeleton tracking