Randomize at the unit where the effect occurs, or the observations are only pseudo-independent

What it is

Whatever layer is randomly assigned is the layer that supplies independent pieces of information. When a household, class, session, or firm is assigned as a block, and the people inside it are treated as unrelated samples, the result is pseudo-independence (pseudoreplication): nominal n inflates, standard errors shrink, and significance becomes too easy. The randomization unit has to match the unit at which the effect happens—if interference lives in team work, assign teams; if the outcome lives on an account, do not test every page view as independent. This is not about whether a control arm exists, and not about whether an A/B test can only choose among built variants: the broken piece is the independence assumption.

Why it happens

A cluster shares devices, social influence, inventory, and unmeasured background, so observations inside it are correlated. When the coin is tossed at the cluster, the number of independent tosses equals the number of clusters, not the number of heads. Treating people as independent counts the same toss many times. Interface logs step on this especially: each click in a session, each day of an account, each member of a company look like many rows and carry about one row of information. If the effect spills into the control (a shared account sees the other version), independence and control purity fail together. Clustering corrections, tests at the randomization layer, or a mixed model that matches the design are ways of admitting how many tosses there were, not of “losing sample.”

Studying it

Declare randomization unit, analysis unit, and why they line up. If measurement must sit at a finer layer (person, session), the plan states clustered errors, the assumed intra-cluster correlation, and how effective sample is derived from the cluster count. Randomization checks run on clusters, not on expanded rows. Simulate: under no treatment, keeping the observed clustering, does the nominal rejection rate still sit near the chosen α? If it overruns, pseudo-independence is already in the test. Report cluster count and row count together; never row count alone.

Where it stops holding

When intra-cluster correlation is near zero—no shared device, no talk, no contested resource—analyzing people costs less, but the plan still has to show that the correlation is negligible rather than switching stories after a non-significant cluster test. Individual-level randomization with spillover is interference, not pseudoreplication, and needs another design (cluster randomization, isolated markets). Rolling many page views up to the account before analysis throws away within-time structure and repairs one layer of pseudo-independence; which to choose depends on the layer at which the effect is defined.

Applying it

• First line of the experiment ticket: randomization unit (user / account / session / team / company). Analysis may not assume independence finer than that layer.
• Results tables list cluster count and row count; a draft with only rows goes back.
• If a click-level test is significant and the account-level test is not, take the account-level conclusion; do not write the click layer into the ship note.
• Check: collapse every account to one row and rerun. If the call flips, the original significance was pseudo-independence.

Related

• Same group: Q3.18.1 Without a control, treatment and time trend cannot be separated · Q3.18.3 Within-subjects designs need order handled by counterbalancing · Q3.18.4 A group gap is a treatment effect only after other variables are controlled
• Adjacent: Q3.04 A/B testing · Q3.11 Log and instrumentation analysis
• Search terms: unit of randomization · pseudoreplication · clustered standard error