Multivariate tests examine several factors and their interactions at once

What it is

Factorial multivariate testing crosses two or more factors in one experiment so every level of each factor appears with every level of the others. The target is not only whether a button change helps, but whether that effect reverses under a different headline, price display, or default. This is not an A/B split between two complete packages: that contrast compares whole treatments and cannot attribute a difference to a named factor, let alone test whether factors depend on one another.

Why it happens

In a balanced crossed assignment, each observation contributes to every main-effect estimate, and factor effects can be separated when the design is orthogonal. An interaction is the leftover pattern after additive main effects: the simple effect of A at one level of B differs from the simple effect at another. Sequential one-factor A/B tests treat the rest of the interface as a fixed background and therefore estimate a conditional effect in that background. Once background and treatment change together, the difference could be the new factor, time drift, or their combination. Only a crossed cell structure makes those combinations comparable.

Studying it

Independent variables are the factors and the cells they form. The dependent variable is a prespecified primary metric, with process metrics used only to inspect mechanism. Fit a model that can estimate interaction terms rather than a disconnected set of pairwise cell tests. Document the interface state of each cell and verify that implementation did not fuse two factors. A dry run checks that every cell receives traffic and that randomization is at the intended unit. If higher-order interactions are declared ignorable in advance, write that as a residual hypothesis to be checked, not as a term deleted after seeing results.

Where it stops holding

When levels cannot be manipulated independently—dark theme forcing button contrast, for example—the cells are not orthogonal and the interaction absorbs an implementation constraint rather than a user response. Personalization, frequency caps, and mutually exclusive features break random crossing. If the decision is simply which of two complete packages to ship, a package-level A/B test is the direct contrast; forcing a factor split adds nothing. Laboratory factorials control materials; live traffic still carries novelty, learning, and contamination, so estimable cells do not imply generalizable effects.

Applying it

• Put into one round only those factors that can ship independently and that might actually depend on each other; leave the rest for later one-factor tests.
• List a shippable interface state for every cell and screenshot-check that factors are not fused.
• Keep a single primary metric; write before launch whether an interaction can change the shipping rule.
• If the result can only say which full combination won, report that contrast and do not claim a factor decomposition.

Related

• Same group: Q3.05.2 Required sample size grows sharply with the number of factors · Q3.05.3 Interaction effects are harder to interpret than main effects
• Adjacent: Q3.04 A/B testing · Q3.18 Experimental design and controls
• Search terms: factorial multivariate testing · interaction effect · crossed design