With a large enough sample, tiny differences become significant
Aliases: excess power · trivial effect · large-N significance
What it is
A hypothesis test gets more sensitive as the effective sample grows. With enough traffic, a few tenths of a point in conversion or a fraction of a second in completion time can clear a conventional significance cutoff. That call means the estimate is precise, not that the difference matters to people or to the business. Large samples pry “detectable” apart from “worth caring about”: the easier significance becomes, the less it can stand in for magnitude.
Why it happens
Standard error falls roughly with the square root of independent observations. Any mean or rate difference that is stably not zero will be rejected once N is large enough. Interface logs, full-population experiments, and long windows cross that threshold readily. Real but trivial gaps—an extra idle click, a millisecond reading-time shift from word order—then receive the same “significant” label as a serious barrier. Under large N the label’s information collapses toward “the sign is probably not zero,” which adds almost nothing to action.
Studying it
Plan for the precision needed to see the smallest effect you care about, not for endless significance. Report N and standard error beside the effect so a reader can see whether the star came from magnitude or from sample size. For already oversized samples, pre-declare that effects below the action threshold are trivial even if significant, and keep them out of the main claim. Equivalence or minimum-effect tests ask whether one can reject “the effect is too small to matter,” rather than only whether one can reject zero. When the log is a stream of events, account for clustering by session so correlated clicks are not treated as independent fuel for sensitivity.
Where it stops holding
In safety-critical or fairness settings, a stable tiny harm can accumulate into something unacceptable; the argument then runs through cumulative risk, not through “it was significant, so rewrite the label.” Small-sample studies do not have this problem; their problem is that even the direction is unstable. When the effect itself changes with mix (peak hours blended with troughs), a large-N significant result is an average over a mixture that may describe nobody’s experience.
Applying it
- Write in the experiment doc that differences below a stated magnitude are not a reason to change the product, significant or not.
- Lead large-traffic summaries with magnitude and interval; park significance in a note.
- When a result is significant but below the pre-committed threshold, close it as trivial; do not reshape the interface just to absorb the star.
- Check: recompute with one-tenth the sample. If “must change” becomes “cannot see,” the original conclusion was tied to N, not to magnitude.