A/B tests compare built variants; they do not invent new ones

What it is

An A/B test is a choice among already implemented candidates: two (or more) pieces of code, copy, or layout enter a random split, and one is declared better on a specified metric. It does not produce a third structure that was never built, and it cannot translate “better” into “what is still missing.” The experiment ends with a ranking on the candidate set, not with need finding. Treating the test calendar as the innovation process trains a team to invent only what can be shipped as a variant, because ideas that were never built never enter the comparison.

Why it happens

A random split requires each arm to be a deliverable product state; otherwise people assigned there hit an unfinished path. That constraint locks search onto points whose engineering cost is already paid. Generating a new option depends on observing failure reasons, unmet goals, and alternatives—material from interviews, fieldwork, and exploratory prototypes, which does not grow out of a metric gap. The gap only answers “which of the existing points is higher.” The more homogeneous the candidates, the “cleaner” the experiment, and the less likely a structurally different option is to appear. The method then runs more often while the option space narrows.

Studying it

Code a period of shipped experiments by structural distance among candidates: copy-level, component-level, flow-level, task-model-level. Count volume and win rate at each band. If almost all are copy-level, the generative step was skipped; the experimental method did not fail. Trace winning ideas to their source; if every source is a tweak of the previous winner, record whether search is contracting. A contrast is to run small-sample generative work before A/B on the same question, versus throwing two existing drafts straight into traffic, and to see whether the shipped solution still sits inside the original candidate set.

Where it stops holding

When the question is already reduced to two implementable, structurally complete options (two lawful defaults, for example), selection is the whole job and A/B need not generate. Exploratory multi-arm tests that force an unfinished concept into live traffic harm users; they do not lift the “does not generate” limit. Schemes from expert review or design workshops still have to become runnable arms before the split; an unrealized sketch has not been compared inside the experiment.

Applying it

• On the experiment ticket, list the candidates and the questions that are not on that list. The latter go to generative research, not to an experiment slot.
• Do not replace problem definition with “ship two versions and test.” Without a success criterion independent of the two drafts, do not split traffic.
• Each quarter, inspect the mix. If nothing above flow-level is in the candidate set, pause testing and add observation and prototypes.
• When archiving a winner, attach the alternative structures that were never tested, so the next round does not only fork the winner.

Related

• Same group: Q3.04.1 Random assignment is what licenses a causal reading · Q3.04.3 Local A/B optima can hide structural problems · Q3.04.4 A short-horizon lift can be long-horizon harm
• Adjacent: Q2.13 Co-design workshops · Q1.01 Framing research questions
• Search terms: variant set · generative versus evaluative · A/B selection