The same input can yield different outputs

What it is

Same prompt, same model, same session settings — press Generate twice and you can get two poems, two summaries. That is not a cache miss and not a network blip. The process itself is sampling. Output stochasticity means the conditional distribution contains many legal next tokens; each run takes one path.

Keep this apart from “the model was updated.” Version and parameters can sit still and the path can still fork. The user is not calling a function. They are drawing a sample.

Why it happens

Autoregressive decoding scores the vocabulary at every step, then draws the next token under temperature, top-k, nucleus, or a cousin of those rules. With temperature above zero, low-probability tokens remain reachable. Even at temperature zero, batching, non-associative floats, and speculative decoding can split two “greedy” traces. “Same input” does not mean “same trajectory on the same graph.”

People read input–output as a function: same argument, same value. Sampling breaks that default. It does not mean the model is off. A polished result is one point drawn from a distribution; the next draw can be a neighbour that is not the same point.

Studying it

Freeze prompt, model version, and decoding parameters. Sample the same input n times and measure spread: n-gram overlap, embedding cosine, variance on the task metric, intra-class correlation of human ratings. Independent variables: temperature, nucleus threshold, speculative decoding on or off. Dependent variables: surface difference, semantic difference, whether task success stays stable.

Split “literally different” from “a different answer for the user.” Two emails that open 您好 versus 你好 may be equivalent; one that promises a refund and one that only apologises are a decision-level split. BLEU alone will count the first as instability and under-count the harm of the second.

Where it stops holding

Calls pinned to a closed answer set — retrieval, classification, a calculator — have their randomness squeezed by downstream checks, so users barely see it. In image or music tools the spread is often the point, not a mismatch. Internal pipelines that lock temperature and pin a seed can look reproducible inside one version, then split across machines or batch sizes. This entry is the fact that one input names a distribution. It does not say how the UI should disclose that, nor how a lucky hit gets read as skill.

Applying it

• Near the generate control, say in plain language that the same request can come back different. Do not dress the button as Query or Calculate.
• When stability is required, offer a deterministic path: turn sampling off, pin a seed, or route through rules or retrieval. Do not verbally promise “we will try to be consistent.”
• Make “generate again” an explicit further sample that keeps the previous result, rather than overwriting it.
• Check: same account, same prompt, three runs in a row must be allowed to differ; if the product claims stability, the three must match byte-for-byte. Put that check on the release list and rerun it when the model or decoder library moves.

Related

• Same group: L1.01.2 Interface conventions assume actions are repeatable and results are stable · L1.01.3 Users misread a lucky correct answer as stable competence · L1.01.4 Controls promise that the same action yields the same result; generation breaks that promise · L1.01.5 Retry cannot tell whether the phrasing was wrong or the system itself is fluctuating · L1.01.6 Undo and redo lose their meaning when output cannot be reproduced · L1.01.7 Presenting regeneration as “refresh” implies the previous result was a failed load · L1.01.8 Presenting variability as several parallel options is more honest than hiding it behind a single result
• Nearby: L1.11 Reproducibility of non-deterministic output · L3.01 Generating multiple options · L1.12 Latency and streaming experience
• Search terms: output stochasticity · sampling variability · non-deterministic generation