Random assignment is what licenses a causal reading

What it is

An A/B test randomly assigns users (or sessions, or accounts) to an already built variant A or B, then compares a pre-specified outcome. The split is not there to “show the new feature more fairly.” It balances, in expectation, both observed and unobserved confounders—concurrent campaigns, season, people who opt into novelty—so a difference in outcomes can be attributed to the difference in variants. Without randomization, people who click into a new version are often already more active; the observed lift can be entirely compositional. Causal inference here hangs on the assignment mechanism, not on a later model that “controls for a few variables.”

Why it happens

Each unit enters an arm with a known probability, so background features on the two arms converge over the long run. Unmeasured temperament, aging devices, and hidden motives are scattered by the same mechanism; they need not be listed in a regression. That is unlike “two groups that look similar”: matching or stratification can balance dimensions someone thought of; unthought dimensions can still bias. Failed randomization reverses the story: if implementation dumps all new users into B, or if members of one household see different versions, balance is broken and the causal chain snaps at assignment. What can be attributed is the variant assigned, not the variant finally used—the latter already mixes in compliance.

Studying it

Audit the assignment log: assignment occurred before the outcome window, arm sizes match the intended probabilities, and key covariates show no large post-assignment holes. If a hole appears, inspect implementation (hash salt, caches, login state) before repairing the experiment into an observational study with post-hoc matching. Compare arms as assigned (intention-to-treat), and leave people who never opened the new version in their assigned arm. Pre-register the primary outcome so a causal story is not chosen after scanning many metrics for a significant one.

Where it stops holding

When units interfere (shared accounts, social referral, limited inventory), treatment in one arm leaks into the other and randomization no longer isolates interference. After the test ships to everyone, a historical contrast is not a second randomization. Comparing people in natural traffic who “used the new feature” with those who did not is not a random split, even if the counts look even. Randomization licenses internal causality; it does not by itself license the same effect in another country or next quarter.

Applying it

• Dry-run the assignment log before launch: probabilities, deduplication, and “assign first, then count outcomes.”
• Analyze the primary outcome by assigned arm. Treat “actually saw B” as a compliance description, not a replacement for the main comparison.
• If a segment sits almost entirely in one arm, stop reading causality; fix the split and rerun.
• The report must name the randomized unit and the outcome window. A number that cannot show assignment before outcome is not an experimental result.

Related

• Same group: Q3.04.2 A/B tests compare built variants; they do not invent new ones · Q3.04.3 Local A/B optima can hide structural problems · Q3.04.4 A short-horizon lift can be long-horizon harm
• Adjacent: Q3.18 Experimental design and controls · Q3.05 Multivariate testing
• Search terms: random assignment · A/B test · causal inference