Testing many metrics at once raises the chance of a false positive

What it is

An experiment that significance-tests conversion, time, ratings, clicks, retention, and more, each at nominal α, can call pure noise an effect on every one of those tests. As the number of tests grows, the probability of at least one false positive—the family-wise error rate—grows with it. Five independent 0.05 tests, all noise, still have about a 1 − 0.95⁵ chance of “at least one significant,” well above 5%. This is error-rate inflation from the count of metrics. It is not yet picking a cell from audience breakdowns, and not yet the procedure of correcting a threshold or naming a primary.

Why it happens

Each test keeps its own tail probability under its own null. When those nulls are jointly true, repeated rejections form a union: more metrics, more chance that one of them clears the line. Interface experiments are rich in metrics, often correlated, so the inflation factor is not a simple product of the count, but the direction is unchanged—more tests, more false finds. A team that then “talks about whichever was significant” makes a second selection on an already inflated list, and there is always a story. A dashboard painting dozens of results green looks like a harvest and is statistically many coin tosses. The error rate belongs to the whole table, not to the one row that gets narrated.

Studying it

List every metric that will be tested in the analysis plan, and compute or simulate the probability of at least one significant result under a global null. Report the number of tests and the number of uncorrected significances together. Shuffle treatment labels and rerun the whole table; if the nominal hit rate exceeds α, inflation is already in the procedure. Highly related metrics can be answered with one multivariate test or a joint model rather than ten univariate tests treated as unrelated. Split exploratory and confirmatory metrics onto separate tables; only the confirmatory table’s error rate is read as a family.

Where it stops holding

Showing many metrics descriptively, without tests, does not create a family-wise error rate, provided the language stays descriptive. Strongly correlated metrics (adjacent funnel steps) inflate less than independent ones, and still not by zero. When every item on a safety or fairness list must pass on its own, the thing being controlled is missed harm, and a higher false-positive rate can be an explicit trade—not a pretense that there were not multiple tests. An experiment that tests one pre-registered metric does not inflate at this layer.

Applying it

• The top of the results page states “this experiment tested k metrics”; a draft without k goes back.
• Do not summarize as “three of them were significant, so the change works”: those three came from a longer table.
• Metrics that never entered the test list must not receive stars on the dashboard.
• Check: shuffle treatment labels and rerun the same table. If stars still appear, the original stars are not findings.

Related

• Same group: Q3.22.2 Correct the threshold or name a primary in advance · Q3.22.3 Reporting the significant slices is data peeking · Q3.22.4 The primary is locked before launch, not chosen after
• Adjacent: Q3.21 Statistical significance and effect size · Q3.05 Multivariate testing
• Search terms: family-wise error · multiple endpoints · false positive