A single comparison cannot rank two options; the conclusion must rest on many draws

What it is

A product runs “new prompt” against “old prompt,” or model A against model B, once each, and writes the winner as an improvement. Once against once compares two random numbers. One draw cannot rank two systems: a ranking is a claim about distributions, and the claim has to come from many samples.

Side-by-side for a user picking a subject is a preference about this instance. Using that same side-by-side to announce which system is better steals an evaluation task from a choice task.

Why it happens

Two high-variance distributions, one point each: the order of the points can oppose the order of the means. Variance of generate quality is usually not small; one lucky draw is enough for the worse system to win. Launch events, A/B reviews, prompt tuning that stop at n=1 are optimising luck. Statistically this is the shallowest sampling error, ignored because interfaces are good at one juxtaposition.

People then read one win as a stable edge, and later decisions (switch models, change the default prompt) lock onto that misjudgement. Non-reproducibility in evaluation is treating n=1 as n=∞.

Studying it

Take two emitters with known mean gap and variance; simulate the product’s current “run once and vote.” Measure how often the vote picks the better mean. Then raise n to 5, 11, 30. Independent variables: variance, true gap, blind order or not. Dependent variables: pick-right rate, decision regret (long-run metric drops after the switch).

When the true gap is small, n=1’s pick-right rate is a coin. That is the figure to take to the review meeting.

Where it stops holding

A user picking a subject or a composition in one session is right to use n=1, because the object is this instance, not the system. Early exploration using one juxtaposition to find a spark should not be treated as a benchmark either. Automated eval that already aggregates many samples inside each request is n>1 behind a surface that looks like once — write that in the method, or others will think you also compare on a point. This entry does not treat what a trace must store.

Applying it

• Any comparison that will announce A beats B must write n and the aggregation. n=1 does not enter a release note.
• Review surfaces default to paired repeats, or sample then blind-rate, not one pretty picture each.
• Split “the user picked this one” from “the system is better.” Do not use counts of the first as evidence of the second.
• Check: take the last improvement you announced and look at its n. If n=1, downgrade the claim to “once it looked better.” Then draw 10 times each on the same prompt and see whether the order holds — if not, that comparison was luck.

Related

• Same group: L1.11.1 Stochastic defects will not replay · L1.11.2 Seeds do not survive version changes · L1.11.3 Non-reproducibility flattens the learning curve · L1.11.4 Trace needs input, version, and output together
• Nearby: L3.07 Multi-option generation and side-by-side comparison · L3.13 User feedback loops on generation quality · L1.01 Mismatch between stochastic output and deterministic UI
• Search terms: n greater than one · single-draw comparison · evaluate distributions