Generation latency usually sits in the seconds people can feel waiting but will not yet abandon

What it is

Press the button, and the first words often arrive between a couple of seconds and a dozen. Shorter than about a second, waiting is not yet an event; longer than about ten, people look for something else to do or leave. Generation happens to sit in the band where waiting is felt and not yet abandoned. The band decides the kind of experience problem: not as large as “do we need a progress bar,” not as small as “imperceptible,” but a blank that has to be accounted for.

This entry only locates which band the delay sits in. How streaming changes perception, and how a half-product invites judgement, come after.

Why it happens

People classify blanks roughly by order of magnitude: instant, a bearable pause, a wait that needs explaining, a wait that licenses leaving. Classic latency work tied those bands to forms of feedback. Generation moves compute from local milliseconds to remote sampling, and the mean falls between the second and third bands, with large variance — sometimes 1.5 s, sometimes 20. The same control cannot stably belong to one band, so it cannot live on one feedback form.

Sitting in the band has a social effect too: people start attributing the blank (broken? thinking? lazy?) before any word appears, and that attribution already prices this generation.

Studying it

Measure your product’s time-to-first-token and total-time distributions, overlaid on an abandon curve. Independent variables: live “writing” feedback or not, a coarse remaining magnitude or not. Dependent variables: abandon, attribution (fault / thinking / normal), second clicks in the blank.

Report the distribution, not only the mean. A mean in the band and a tail in the abandon zone — experience is the tail.

Where it stops holding

A local small model or a cache hit falls into instant, and band-level design becomes noise. Batch and overnight jobs fall out of the band and should be built as background work, not squeezed into a chat bubble. A weak network stacks network delay on generation delay; attribution will point at the net, not the model, and feedback has to be able to say them apart. This entry does not discuss whether to stream.

Applying it

• If first token is still missing at one second, give a live “generating,” not a blank spinner until abandon.
• Pick feedback from the distribution, not the mean: if P95 is already out of the band, there must be a way to leave or to see progress. Do not assume everyone sits on the mean.
• Do not treat a second click as a new stacked request; extra clicks in the band are mostly anxiety, not a new intent.
• Check: plot the first-token histogram against abandon times. If the belly is 2–10 s and you have no live feedback before 8 s, the blank is unclaimed. Then see whether P95 already belongs to a background job.

Related

• Same group: L1.12.2 Streaming shortens time-to-first-token, not duration; it changes perception, not fact · L1.12.3 Token-by-token invites judging before completion, so decisions rest on a half-product · L1.12.4 Mid-stream self-correction shows content that will later be overturned · L1.12.5 Long background jobs should not stream; they need progress and permission to leave
• Nearby: L3.11 Streaming the generation process · I2.07 Perceived performance · L1.06 Graceful degradation of AI failure
• Search terms: felt-wait band · time to first token · generation latency