Calling a result significant means little until the chart also says which statistical test produced it

What it is

The marker "statistically significant" omits a crucial premise: significance is relative to some test. The same data may yield different p-values under a t-test, a Mann-Whitney test, or a chi-square test—not because one is "more correct," but because they make different assumptions about the data (normality, independence, continuity) and test different hypotheses. An unlabeled significance marker is an incomplete evidence statement: readers cannot assess under which assumptions the conclusion holds, or whether a reasonable alternative test would reach the same verdict. When different charts use different tests yet the same stars, readers cannot distinguish the methodological differences behind those "significant" labels.

Why it happens

The test determines the p-value because each test makes a different choice about "how to measure extremeness": parametric tests (t-test) assume a specific distribution and gain power from that assumption; nonparametric tests (Mann-Whitney) avoid distributional assumptions at some cost in power; chi-square tests for proportions answer a different type of question than t-tests for means. When the data violate the test's assumptions (e.g., heavy-tailed data under a t-test), the p-value's computational foundation fails—the same data may cross or miss the threshold depending on whether the test is appropriate. Annotating the method makes the conclusion auditable: knowing which test was used, readers (or reviewers) can assess whether the assumptions are reasonable, whether the sample size suffices, and whether the conclusion is robust to an alternative test. Without the annotation, the audit chain breaks at its first link—"significant" becomes an untraceable authority label rather than verifiable evidence.

Where it stops holding

How much detail to include (test type, sidedness, correction) depends on space budgets: chart space is limited, full statistical details belong in tooltips or footnotes, and the chart itself should at least name the method category ("t-test"). For internal dashboards standardized on one test, a system-level declaration can replace per-chart repetition. Another boundary is "do readers need to understand the test to use the conclusion"—business readers usually do not, but the annotation's purpose is not only reader comprehension; it is traceability when a conclusion is later questioned: three months on, someone asks "how was this significance computed," and without annotation there is no answer.

Applying it

• Note the test method in the tooltip, footnote, or caption of every chart annotating significance (e.g., "Mann-Whitney U test, two-sided").
• Bind test selection to data type at the system level (chi-square for proportions, t-test for means, Poisson-family methods for counts) so chart makers cannot hand-pick inappropriate tests.
• Keep version history when the method changes so conclusions from different periods of the same chart map to different methods.
• Verification: sample 3 starred charts at random and check whether the test used can be determined from the chart or its notes; any single failure means annotation is missing.

Related

• Same group: U8.03.1 Significance markers do not convey the size of the effect · U8.03.2 Uncorrected multiple comparisons inflate significance markers systematically · U8.03.3 There is no visual cliff between significant and non-significant · U8.03.5 Showing effect sizes with intervals conveys more than significance labels
• Nearby: U8.02.1 Whether an error bar shows standard deviation, standard error, or confidence interval must be stated · U8.03.1 Significance markers do not convey the size of the effect
• Search terms: statistical test annotation · test selection · reporting standards