Pinning a random seed only reproduces inside one version; it dies when the model or the prompt is updated

What it is

Log the seed and you think you have the random seed from a lab notebook. Change the weights, the decoder library, even the system prompt, and the same seed walks another trajectory. A seed is version-local: it reproduces sampling on this build, not the same experiment across time.

Support believes “we have a seed, so we can replay.” The day after a release that replay may already be a lie.

Why it happens

A seed only enters the PRNG. Everything downstream — weight quantisation, batch size, speculative decoding, a safety layer rewriting the system prompt — changes the conditional distribution, and the seed then draws from another distribution. Product release treats those downstreams as routine; the seed is stored on tickets as a long-lived identifier. The two lifetimes do not match.

The prompt side is more hidden. The product wraps the user’s sentence in an iterating system prompt and tool spec. The user thinks the input did not change; the seed is drawing against a complete prompt that did. If the log stores only the user-visible sentence, the seed cannot even save you inside a version.

Studying it

Pin a seed and change, separately: a minor weight revision, the decoder, one sentence of system prompt, batch size. Measure verbatim recovery. Independent variables: kind of change, whether a full prompt snapshot is bound to the seed. Dependent variables: identity recovery, semantic recovery.

“Semantically close” is not reproduction. In a defect setting, reproduction must surface that error again; close will smear it away.

Where it stops holding

A fully self-hosted, frozen image with no hot updates can use a seed as a replay key for the life of the image. Swap the image, the key is dead, and the release note must say so. In image tools a seed is often a user-facing parameter and is not expected to hold across models; say that on the control. This entry does not treat which three things must be stored together to trace. It treats that a seed alone is not enough, and especially not across versions.

Applying it

• Beside the seed, mark model version, decoder, and a hash of the complete prompt. Missing one, the UI may not say reproducible.
• Replays of old tickets after a release must be labelled “indicative only; this version does not guarantee the same trajectory,” not pass/fail verdicts.
• A user-visible “again with the same seed” promises the current version only. When the version moves, stop the control or change the copy.
• Check: ship a minor version that only changes one system-prompt sentence, replay a known output with the old seed. If the product still shows Reproduced and the bytes differ, the seed has been treated as cross-version magic.

Related

• Same group: L1.11.1 Stochastic defects will not replay · L1.11.3 Without reproduction people cannot form a stable mental model · L1.11.4 Trace needs input, version, and output together · L1.11.5 One draw cannot rank two systems
• Nearby: L1.01 Mismatch between stochastic output and deterministic UI · L1.02 Expressing capability boundaries · L4.15 Accountability and traceability
• Search terms: seed is version-local · reproducibility across releases · prompt snapshot