Different outputs from the same input make defects hard to reproduce; a user-reported problem may not replay

What it is

A user pastes a wrong screenshot; the ticket says “I typed this sentence.” Support runs the sentence again, gets another passage, the error is gone, the ticket is marked unreproducible. The error may still live in the distribution. Stochastic defects will not replay on demand: the report describes one draw; the reproduce button describes another.

This is not the user-facing side of “outputs vary.” It is defect management being taken apart by sampling.

Why it happens

Classic defects assume failure is a function of state. Given input and environment, failure should recur. Sampling turns failure into an event on a distribution; the chance of a successful replay is that event’s mass, which may be a few percent. Support then treats one missed replay as “user environment” or “already fixed,” and a rare, expensive miss starves in the queue.

Users are frustrated the same way: they are asked to “try again so we can see,” draw a normal point, and feel disbelieved. Report cost rises; later they report less.

Studying it

Plant a defect known to appear with probability p, and run a real ticket flow: user reports, support reruns n times. Dependent variables: share of tickets closed unreproducible, time from report to being treated as a real defect. Independent variables: n, whether the original output is kept, whether a ticket may be opened on the original output rather than the original input.

p must be low. A high-p plant makes the classic flow look sufficient.

Where it stops holding

An internal pipeline with temperature locked and seeds logged can replay inside one version — not across versions, which is the next card. Structural errors a determinate checker catches do not depend on replay. When the user has already pasted the full output, what is missing is not replay but taking that output as evidence. This entry is only the failure of “run the input again” as the condition for opening a case.

Applying it

• The primary key of a case is that instance’s output (or its hash), not the prompt text. If output was not saved, ask the user to paste the result before you rerun and close.
• Support reruns are at most a sample of “is this still frequent,” never a verdict of “did this exist.”
• Give users “report this result as a problem” without requiring them to reproduce it.
• Check: hand support a known-bad output and no seed. If the ticket closes because “I cannot get it here,” the flow is using the wrong definition of reproduce. Then count closures for unreproducible that later reappear as the same kind of output.

Related

• Same group: L1.11.2 Pinning a seed only reproduces inside one version · L1.11.3 Without reproduction people cannot form a stable mental model; the learning curve flattens · L1.11.4 After-the-fact trace needs input, system version, and output kept together · L1.11.5 A single comparison cannot rank two options; the claim must rest on many draws
• Nearby: L1.01 Mismatch between stochastic output and deterministic UI · L3.13 User feedback loops on generation quality · L4.15 Accountability and traceability
• Search terms: non-reproducible bug · stochastic defect · replay by prompt