A significant result is not by itself a reason to change the product
Aliases: practical significance · decision threshold · SESOI
What it is
Statistical significance says only that, if a null were true, a difference this large or larger would be uncommon. It does not say the change is worth making. A copy experiment can clear a pre-set threshold and still move a handful of conversions while costing engineering, support, accessibility re-checks, and brand risk. Treating significance as a ship license mistakes a rejection rule for a decision rule.
Why it happens
A test asks whether a difference looks like noise. A decision asks whether, given cost, reversibility, opportunity cost, and who is harmed, one should act. Both questions read the same numbers under different standards. High cost, asymmetric harm, or hard rollback can make even a non-noisy difference not worth shipping. Very cheap, clearly directional changes can be worth a limited try even when they miss the threshold. The significance cutoff is a convention, not a loss function. Writing “significant” into a launch checklist replaces that tradeoff with a switch that ignores cost.
Studying it
Before seeing results, write the smallest effect size of interest: how large a gain must be to justify this change’s cost and risk, and how large a harm would block launch. The analysis plan states both the test and the decision: significant-but-below-threshold does not ship; an interval sitting entirely on the harmful side stops the line rather than “running until significant.” Put engineering hours, support load, migration cost, and irreversibility in as explicit inputs, not after-the-fact arithmetic. What later work should reproduce is the decision rule, not merely whether p again fell below a cutoff.
Where it stops holding
Hard constraints in safety, regulation, or accessibility can make the mere presence of a difference the decision object; significance then behaves more like a compliance check, and the direction and magnitude still need to be reported. An exploratory finding that happens to be significant does not retroactively supply an action threshold. When the real cost is political or sunk, and not user harm, using significance to end debate is especially dangerous: it offers a ceremony for stopping thought, not a tradeoff.
Applying it
- At kickoff, write how large an improvement must be to justify the change, and how large a regression must be to stop.
- On the results page, show the significance call, the magnitude, and the pre-committed threshold together; never post only “significant, recommend full rollout.”
- When a result is significant but below the action threshold, record it as “detectable, not worth doing”; do not move the threshold to match a desired launch.
- Check: ask a colleague who did not run the experiment to decide from magnitude and cost with p hidden; if that judgment systematically reverses after they see p, significance is being used as the decision.