Interaction effects are harder to interpret than main effects

What it is

An interaction means the effect of one factor changes with the level of another, not that “both factors matter.” A main effect is still a marginal sentence—“A is better”—averaged over the other factors. An interaction forces the sentence “whether A helps depends on B.” Reading it therefore cannot stop at a significant coefficient; it has to return to simple effects and cell means. Crossover interactions are especially easy to miss: the margin looks flat while combinations point opposite ways.

Why it happens

An additive model treats each factor as a shift that can be slid across the other. Interaction is departure from that additivity: the same button may raise conversion under urgent copy and lower trust under neutral copy. Averaging cells over B lets opposite simple effects cancel, so the main effect of color sits near zero and the report says “color does not matter”—the decision-relevant pattern has already vanished. A three-way term further requires “it depends on the combination of the other two.” That slice is hard to compress into an executable rule, so teams slide into “everything matters,” which is not an interpretation.

Studying it

Plot cell means and simple effects before the interaction term. Prespecify which kind of interaction would change the shipping rule: same-direction amplification, attenuation, or reversal. For reversals, estimate simple effects and their intervals at each level rather than quoting only an interaction p-value. Treat higher-order terms as noise unless a mechanistic slice was named before launch. Recheck the chosen simple effect on held-out traffic or in a follow-up one-factor test.

Where it stops holding

Difficulty of interpretation is not a reason to ignore interactions. When the product logic is already conditional—new and returning users see different defaults—the simple effect is the target and the marginal main effect should not be the headline. Exploratory screening of many interactions will turn noisy slices into stories. When factor levels are convenient codes rather than meaningful states, an interaction may only reflect an implementation glitch. A statistically significant interaction with a tiny effect rarely justifies a branched product.

Applying it

• Lead the readout with a cell table or interaction plot, then one sentence of the form “at which level of B, A moves which way.”
• If simple effects reverse, do not ship the “better on average” level; branch by condition or drop the factor.
• Do not put three-way or higher interactions into a business readout unless a mechanism was declared in advance and cell n can support it.
• Re-test the chosen conditional effect on its own; withdraw the combination story if it does not replicate.

Related

• Same group: Q3.05.1 Multivariate tests examine several factors and their interactions at once · Q3.05.2 Required sample size grows sharply with the number of factors
• Adjacent: Q3.22 Multiple comparisons and result picking · Q3.13 Statistical significance and practical importance
• Search terms: interaction effect interpretability · simple effects · crossover interaction