Strong backlight or overly dark environments significantly reduce skeleton and hand-keypoint accuracy

What it is

When someone stands in front of a window, a lamp at their back, or the room is down to a night light, skeleton and finger keypoints fall from usable to jittering or gone. Backlight cuts the body to a silhouette with no palm texture or finger gaps; too dark maxes exposure and lets noise eat edges. These are two specific extremes that drop accuracy hard, not a vague “needs good light.” The general dependence on intensity and direction sits one layer up; this is how those two ends ruin keypoints.

Why it happens

In backlight, auto-exposure either protects the window and crushes the person to a black slab, or protects the face and blows the window to a white hole; the hand often lands in the unusable band between. A silhouette has no local features for keypoints; the model starts calling a shoulder an elbow and a cuff a palm. Too dark is not enough photons: gain rises, exposure lengthens, motion smear turns a pinch into a blob. IR depth usually still lives in dark visible light, and dies in solar backlight when IR saturates. Accuracy drops before total loss: scores still emit, coordinates are already untrustworthy, continuous adjustment shakes. What users see is “the hand is still in the picture, the slider is wandering.”

Studying it

Make repeatable backlight with a window or a large lightbox, and too-dark with dimming. On the same skeleton/hand model, measure keypoint error against ground truth (mocap or labels) in pixels or millimetres, detection rate, and jitter. Report RGB and IR separately. Add “backlight plus a fill lamp toward the face” to see whether accuracy can be rescued. Do not stop at “is the gesture still recognized”—keypoints may already be rotten while the classifier is still getting lucky.

Where it stops holding

HDR or active fill can push the two ends outward; they do not make them vanish. A night product that is IR-only no longer cares about dark visible light; solar backlight still matters. Dark skin against a dark ground has worse contrast in low light; a lab sample of light skin will underestimate. Backlight hurts full-body skeletons and hands differently: the body outline may remain after fingers are gone, so a finger vocabulary collapses earlier than an arm-wave one.

Applying it

• In install, avoid a habitual stance square to an unshaded window. If the window is mandatory, add fill toward the user or switch to an IR-primary pipeline.
• When keypoint jitter exceeds the product threshold, stop writing coordinates into continuous controls; treat it as accuracy failure, not as the user shaking.
• Accept pinch and scrub standing at the window and under a night light: keypoint error should still sit in a usable band. Having passed in a side-lit office does not cover these two ends.

Related

• Same group: C4.31.1 Reliability of vision-based gesture recognition varies with ambient light intensity and direction · C4.31.3 Occlusion by obstacles or other bodies interrupts continuous motion tracking · C4.31.4 Environmental effects should be tested with the target scene's real lighting before deployment, not laboratory ideal lighting
• Adjacent: C4.13 Full-body pose and skeleton tracking · C4.23 Handling tracking loss
• Search: backlight · low light · keypoint error