End-to-end latency is a chain; optimizing only helps the longest segment

What it is

From a head movement to a photon in the eye there is not one delay but a sum: sensor sample, transport, application, GPU, scanout, pixel emission. That sum is end-to-end latency, usually measured as motion-to-photon. Work only pays on the current longest segment. Shaving two milliseconds off an already-short app thread is not felt; if display scanout eats most of a refresh period, leaving it untouched leaves the sum where it was.

The topic is how the budget is spent, not the conclusion that this latency must be low.

Why it happens

A series total is the sum of its parts; sensation tracks the total; improvement tracks the bottleneck. Sensor filtering 2 ms, another USB packet 1 ms, simulation 3 ms, GPU 8 ms, queued scanout 8 ms, emission another 2 ms — a little over twenty. Cutting simulation from 3 to 1 barely moves the total; cutting GPU-plus-scanout from 16 to 8 is what makes the world stick to the head.

The refresh period is itself a hard budget: 90 Hz is about 11 ms a frame, scanout and persistence eat a slice, and a slow application can only queue for the next frame. That is queue position, not “the software could still be tighter”. Amdahl is literal here: speed up a non-bottleneck and the speedup is one.

So measurement has to be segmented. A single total sends the team after the layer they know (usually the application), which is often not the longest.

Studying it

Tape a photodiode in front of the lens on a flashing pattern, stamp each stage in the driver (IMU interrupt, submit, GPU fence, vsync), and split one nod into a waterfall. Then lengthen or shorten one stage at a time (holding the others) and see whether detection threshold and discomfort follow that stage.

Independent variables: which stage is stretched, that stage’s absolute duration, whether total duration is held constant. Dependent variables: per-stage time, delay from nod to luminance change, ratings of “the world feels sticky”.

The critical control is same total, different bottleneck. If ratings rise only when the bottleneck stage is stretched, “wrong layer” can be separated from “latency is high”.

Where it stops holding

Cloud render and split compute turn transport into a new bottleneck; a fast local GPU cannot fill network jitter. Optical see-through has no “virtual photon” stage, so the waterfall is a different shape and must not be copied across. Tasks with a fixed gaze and almost no head turn are insensitive to the total, and the bottleneck will not show. Late-latch and asynchronous time warp in the lab lift the application stage off the critical path — the measured bottleneck moves to scanout and emission, which is a budget rewrite the runtime has already done, and results must be labelled as such.

Applying it

• Draw the waterfall before starting work: sensor, transport, CPU, GPU, scanout, emission. Change only the current longest slice.
• Schedule against the refresh period as a hard cap, not against “average fps looks fine”.
• An application-side win that is not the longest bar on the waterfall is not the completion criterion of a latency project.
• How to check: recapture the waterfall after the change. If the total fell and the longest stage barely moved, the wrong layer was edited. If the longest stage shortened and the feel is still poor, look at jitter and dropped frames rather than shaving the same stage again.

Related

• Same group: N1.11.2 Reprojection decouples render rate from display rate · N1.11.3 Dropped frames appear as the world jumping during head motion, not slowing · N1.11.4 Latency jitter is harder to adapt to than mean latency
• Nearby: N1.05 Motion-to-Photon Latency · N1.12 Refresh Rate and Persistence
• Search terms: end-to-end latency · latency budget · motion-to-photon