Running dozens of comparisons without correction guarantees some will look significant by chance alone

What it is

A p-value threshold (e.g., 0.05) controls the false-positive rate for a single comparison. When a chart presents many group comparisons at once (20 metrics, dozens of segment slices), each comparison carries its own 5% false-positive rate, and the overall probability of at least one false positive climbs rapidly with the number of comparisons: with 20 independent comparisons, the chance of at least one "significant" result is about 64%; with 100, about 99%. Significance markers on charts without multiple-comparison correction therefore do not mean "all these differences deserve attention"—some are pure statistical noise scooped up by the noise filter.

Why it happens

The mathematical root of false-positive inflation is probability accumulation: each test's false-positive rate is (approximately) independent, and after n tests the probability of "at least one false positive" is 1-(1-α)^n, approaching 1 exponentially. Dashboard interactivity manufactures multiple comparisons naturally: users freely filter dimensions, switch time ranges, compare any two groups—each operation is effectively another implicit test, yet no system performs any correction. More insidious is the analyst's "garden of forking paths": after seeing the data, analysts decide which groups to compare and which test to use, and that choice itself exploits random variation, biasing the results labeled "significant" toward exactly those comparisons that crossed the threshold. Visually, once a star is placed it is accepted as fact—no reader recalculates "how many of these 15 significant results might be false"—and inflated false positives are fossilized into conclusions by their visual form.

Studying it

The classic way to study multiple-comparison inflation is simulation: run the same comparison pipeline as the real dashboard on null data where "all groups are truly identical," then count the proportion of comparisons labeled significant—theoretically near α, in practice often much higher. In user studies, one can present charts with known correction status (corrected vs uncorrected) and measure readers' trust in the count of significant results and the quality of decisions based on them. A methodological caveat: the "number of comparisons" is itself ambiguous—is it the comparisons visible on the chart, or every combination of filters and comparisons the user executed in the session? This definitional ambiguity means "how many corrections should be applied" has no unique answer, a common reason practitioners avoid correcting altogether.

Where it stops holding

Multiple-comparison correction is not required everywhere: exploratory analysis (hunting hypotheses worth testing further) and confirmatory analysis (final judgment on a preset hypothesis) follow different rules—the former tolerates higher false positives for sensitivity, the latter demands strict correction. Dashboards are mostly exploratory, and holding them to confirmatory standards is overly conservative; but when chart results directly trigger actions (auto-pausing underperforming ads), each chart is a confirmatory judgment and correction is non-negotiable. The choice of correction method (Bonferroni is most conservative; Benjamini-Hochberg controls the false discovery rate more-balanced) also affects how many "significant" markers appear, and no single method is uniquely correct. When comparisons are inherently correlated (adjacent age bands in the same population), the effective number of tests is below the nominal count, and over-correction sacrifices power.

Applying it

• When a chart shows significance markers for 3+ group comparisons, state in the caption whether multiple-comparison correction was applied and which method.
• Dashboards that automatically trigger actions (anomaly detection, auto-pausing) must build correction in and encode the corrected threshold into the alert configuration.
• Exploratory dashboards may skip correction but must tag significant markers "exploratory—needs validation," distinguishing them from confirmatory conclusions.
• Verification: rerun a multi-group chart's comparison pipeline on simulated null data where all groups are identical, and count how many of the chart's significant markers reappear; a rate near α means correction is adequate, well above α means inflation.

Related

• Same group: U8.03.1 Significance markers do not convey the size of the effect · U8.03.3 There is no visual cliff between significant and non-significant · U8.03.4 Charts annotating significance must state the test used · U8.03.5 Showing effect sizes with intervals conveys more than significance labels
• Nearby: U7.04.4 Too many alerts train users to ignore alerts · U8.02.2 Overlapping intervals between two groups do not imply a non-significant difference
• Search terms: multiple comparisons · false discovery rate · Bonferroni correction