Required sample size grows sharply with the number of factors

What it is

In a crossed experiment the number of cells is the product of the factor levels. Each added two-level factor doubles the cells and halves the independent observations in a combination. Estimating interactions stably usually demands more total sample than estimating main effects, so overall traffic can look large while every combination is thin. This is combinatorial growth, not a linear surcharge for “a few extra variants.”

Why it happens

Sampling variance of a cell mean rises as that cell’s n falls. Main effects can still pool several cells; an interaction is a difference of differences and therefore uses less effective information. With the same total traffic, a two-by-two interaction contrast is already noisier than a two-arm A/B test; a third binary factor turns four cells into eight and halves n again, leaving almost no precision for a three-way term. Sparse cells also let a few heavy users or bots pull one combination, a bias that averages out in pooled main effects but is treated as “this version won” when shipping a cell.

Studying it

Specify the highest-order interaction that must be estimated, the smallest effect worth detecting, and baseline rate or variance, then plan power or precision per cell, not against total traffic. Simulations should include attrition, imbalance, and outliers, and should inspect the smallest cell n rather than the average. If traffic cannot support a full cross, drop factors, collapse levels, or declare ignorable higher-order terms before launch; deleting cells afterward changes which effects remain estimable. Report effective n per cell, exclusion rules, and whether the smallest cell fell below the preregistered floor.

Where it stops holding

Covariates, blocking, and continuous factors can reduce required n, but they do not turn eight starved cells into one well-powered experiment. The explosion is milder for non-conversion metrics, strongly correlated repeated measures, or designs that only care about a few prespecified combinations. High-traffic products may still support three or four binary factors; small products and rare conversions can fail at two. An underpowered “nonsignificant interaction” means the interaction was not visible, not that effects are additive.

Applying it

• From the primary metric’s baseline and the smallest lift worth shipping, compute the independent users needed in each cell; cut factors if that number is out of reach, rather than launching and hoping.
• Show total traffic and minimum cell n on the same dashboard; do not rank combinations when the smallest cell is below threshold.
• Refuse requests to “just add a button color” that enlarge the cross without adding window or traffic.
• At the end, list empty and near-empty cells; if the decision depends on them, declare the experiment inconclusive and rerun a smaller cross.

Related

• Same group: Q3.05.1 Multivariate tests examine several factors and their interactions at once · Q3.05.3 Interaction effects are harder to interpret than main effects
• Adjacent: Q1.08 Sample size · Q3.21 Statistical significance and effect size
• Search terms: combinatorial sample-size growth · cell size · power for interactions