Camera position is offset from the eye and needs extra correction

What it is

Reach for the cup at the desk edge and the fingertip lands short or long. Video-see-through cameras do not sit in the pupil; they sit forward and usually outward on the housing. The projection of a near object through the lens is not the projection through the eye. That difference is camera-eye parallax. Uncorrected, the hand–eye loop reaches into the wrong near-field geometry.

Buildings and the horizon barely show it. Desks, tools, and someone else’s hand inside an arm’s length push the error into centimetres.

Why it happens

A pinhole camera projects from the lens nodal point. The eye’s nodal point sits millimetres behind the cornea. A headset parks the cameras on the front shell, so a fixed 3D offset — and sometimes a baseline that is not the wearer’s IPD — sits between them. Disparity between the two projections grows as the inverse of depth: near zero at infinity, large enough at twenty centimetres to shift a whole cup rim.

Correction reprojects the camera image through the eye node, often tracking where the eye actually sits in the eyebox. Wrong depth for that reprojection — a plane assumption, sparse depth, treating everything as far — bends near objects the wrong way. Uncorrected, people treat grasp error as a depth signal and learn a directional compensation. Change the cup or the working distance and the compensation fails.

Studying it

Near-field grasping and pointing alignment: real targets at 20–60 cm, comparing no correction, planar reprojection, and depth-based reprojection. An eye tracker splits mean offset from gaze-dependent residual.

Comparisons: camera translation relative to the pupil, target distance, per-eye reprojection on or off. Records: grasp landing error, whether first contact hits, reports of “can’t quite reach,” whether stereo fusion breaks.

Phone passthrough offsets are larger than headset offsets; phone AR error is not an upper bound for a head-worn rig.

Where it stops holding

Far-field tours and distant labels can ignore the parallax; correction barely pays. Large eye motion inside the eyebox leaves a gaze-dependent residual if calibration used headset geometry once; eye tracking is what closes it. Monocular passthrough has no stereo-baseline problem but still magnifies the near field by shifting the node forward. Wearers with strabismus or a single seeing eye, corrected to a mean IPD, dump the whole error onto the used eye. Optical see-through does not have this camera offset — real light still travels through the eye’s own node — so do not copy a VST calibration checklist onto it.

Applying it

• Within-arm work (assembly, desktop, clinical near field) needs reprojection through the eye node, calibrated at the working distance. Do not ship an “infinity” default.
• When depth is unreliable, move near interaction farther out, or add a real haptic referent (desk edge, jig). Do not let people train a compensation inside the wrong perspective.
• Set the stereo baseline to the wearer’s IPD, not the cameras’ mechanical spacing; when the two disagree, keep the eyes.
• How to check: put equally sized real markers at 30 cm and 2 m and have people put a fingertip on the centre. Far alignment with a systematic near bias toward the camera side means the parallax is still uncorrected.

Related

• Same group：N5.08.1 In optical see-through, virtual content can only add light and cannot truly occlude real objects · N5.08.2 Video see-through re-digitizes the real world, so a virtual object can occlude a real one · N5.08.4 Video see-through still undershoots the eye in dynamic range and resolution; highlights and shadows both lose · N5.08.5 Optical see-through still shows the real world on power loss; video see-through blinds
• Nearby：N5.01 See-through Modes · N5.02 Virtual–Real Occlusion
• Search terms：camera-eye parallax · viewpoint offset · nodal offset