Video see-through is controllable but adds latency and image loss

What it is

Look at a second hand on the wall, then put the headset back on: the hand arrives late. Reality is captured as frames and redrawn on a display. That path is video see-through. Brightness, crop, and overlay can all be rewritten in pixels — that is the control. The cost is that every frame is late, and it has already been sampled, denoised, and compressed. It is no longer the bundle of light that would have hit the retina.

Controllable means the compositor owns the reality layer. It does not mean latency or sharpness can be designed to zero.

Why it happens

Once photons hit a CMOS sensor they travel through exposure, readout, ISP, composition, and scan-out. Each stage is a buffer. A global shutter still waits out a frame; a rolling shutter shears vertical edges during a head turn; temporal filtering paints moving contours into smear. When the head moves, the real world updates after the vestibular signal. That is a motion-to-photon path of its own, stacked on whatever latency the virtual layer already has. People compensate by treating the picture as “the world in motion,” not as photons. Compensation has a ceiling; past it the room floats, edges miss, and the hand closes on empty air.

Image loss is the other exit of the same chain. Optics and the sensor cut resolution, auto white balance rewrites color, compression posters a flat wall. Still frames can hide this. A head turn stacks the artifacts on the delay, and the scene is read as “not my room.”

Studying it

Put a millisecond timer in front of the headset and film both the naked-eye view and the passthrough with an external high-speed camera; align the same event to get end-to-end see-through latency. For image quality, shoot a static side-by-side of the same wall and lamp, scoring noise, sharpness, and cast. Sample discomfort with SSQ or VRSQ in a passthrough-only condition (virtual layer off) so vection from virtual motion does not contaminate the reading.

Independent variables: exposure time, compositor buffer depth, head angular velocity, spatial frequency of the scene. Dependent variables: event arrival lag, grasp landing error, rated float, mean opinion score on still frames.

The point of the measurement is whether a precision alignment task — seating a virtual ruler on a real edge — can be entrusted to passthrough, not whether a satisfaction total can bury the delay.

Where it stops holding

Stand-alone passthrough is already much shorter than tethered camera rigs; shorter is not none. For slow browsing and far viewing, image loss is complained about before latency. Seated, low-head-motion desk work can hide the same delay that appears the moment someone stands and turns. In low light the ISP lengthens exposure, so latency and noise worsen together; numbers from a fluorescent lab do not transfer to a night corridor. Vestibular-sensitive wearers will symptom at delays others still call usable.

Applying it

• Mark operations that must sit on a real edge — seating, measuring, near grabs — as latency-critical. Measure passthrough delay on those paths before committing; a still demo shot is not a substitute.
• Expose brightness and sharpness of the reality layer. Calibrate defaults to the target venue’s lighting, not to the indoor lights in a trailer.
• During turns and fast motion, prefer a timely passthrough frame over extra post-processing in the same frame.
• How to check: have people tap a real thumbtack on the wall through passthrough, then again with the naked eye. A systematic miss in the direction of the head turn is the delay dragging the point. Pair a passthrough still with a photograph and mark posterization and blur that would be rejected.

Related

• Same group：N5.01.1 Optical see-through keeps real light but contrast is limited · N5.01.3 See-through mode sets the ceiling on real–virtual fusion
• Nearby：N5.08 Video See-through and Optical See-through · N5.02 Virtual–Real Occlusion
• Search terms：video see-through · passthrough latency · motion-to-photon