Latency jitter is harder to adapt to than mean latency

What it is

Twenty milliseconds every time, and “twenty milliseconds on average, bouncing between twelve and forty”, are not the same delay. The first is stable; the motor system can predict the next head pose against a fixed lag. The second is latency jitter — the same movement is almost glued to the face on one beat and a half-beat late on the next, and prediction is scrambled. People can adapt to a high but steady mean. They cannot adapt to a ragged timeline under that mean.

The comparison is variance versus mean, not which link in the chain is longest, and not how a wall jumps when a frame is dropped.

Why it happens

The nervous system learns a delay between vestibular signals and visual consequences. A constant delay is like a prism of fixed power: pointing error shrinks over days. A delay redrawn every frame leaves the compensation permanently short; residual error shows up as “the hand cannot catch up” and “the world is sticky, then slippery”. Predictive control handles noise worse than bias: bias can be subtracted, noise is met by turning gain down, so movement becomes conservative and slow.

Jitter usually comes from queues, not from the sensor: a GPU timeout, thermal throttle, a background spike on a core, vsync flipping from “just made it” to “wait the next slot”, and latency jumps by a whole refresh period. The mean still looks acceptable; the tail is already suprathreshold.

So reporting only mean motion-to-photon writes a pass for an experience that is fine, then suddenly not.

Studying it

Hold the mean fixed and build two distributions: constant delay versus jitter around that mean at a given standard deviation (or a two-hump: made-the-frame versus waited-a-frame). Tasks: pointing, tracking a head-referenced target, temporal reproduction.

Independent variables: mean, jitter standard deviation or peak, whether jitter correlates with head speed. Dependent variables: pointing error, variance of tracking lag, discomfort, reports of “it suddenly went sticky”.

A photodiode histogram beats a single mean: read the 95th percentile and the largest jump, not just the expectation.

Where it stops holding

Passive viewing with almost no turning or reaching makes jitter hard to separate from the mean, and the complaint weakens. If jitter is confined inside one refresh period and the display still latches a fresh pose every beat, application frame-time variance is partly hidden — measuring app frame-time variance is not measuring jitter at the eye. Cloud-link jitter and local queuing are not the same order of magnitude, and they do not share one adaptation result. Lab jitter from a constant added delay is denser and easier to learn than a real system’s occasional waited-frame, so extrapolation is optimistic.

Applying it

• Write acceptance on percentiles and maximum frame interval, not on “xx ms average” as a pass by itself.
• Take rare timeouts off the critical path: thermal throttling, shader compiles, and synchronous loads should not appear in a session that has already started turning.
• Prefer being stably one refresh later over flipping between “just made it” and “dropped a slot” — that flip is the jitter.
• How to check: sample motion-to-photon or frame interval for at least a minute or two and plot the histogram. A legal mean with a 95th percentile that spans a refresh period fails as jitter. Run the same movements under constant delay; adapted error should shrink.

Related

• Same group: N1.11.1 End-to-end latency is a chain; optimizing only helps the longest segment · N1.11.2 Reprojection decouples render rate from display rate · N1.11.3 Dropped frames appear as the world jumping during head motion, not slowing
• Nearby: N1.05 Motion-to-Photon Latency · N1.12 Refresh Rate and Persistence
• Search terms: latency jitter · frame-time variance · variable latency