A star marking significance answers whether a difference is real, not how large that difference is

What it is

Stars (*, **, ***) or "significant" labels on a chart answer a binary question: "is the observed difference unlikely to be explained by random fluctuation alone?" They do not answer "how big is the difference"—which is usually what decisions actually need. A 0.1% conversion lift can earn three stars with a large enough sample, yet rolling it out to all users may yield an absolute gain too small to cover the change's cost; conversely, a business-meaningful 5% lift in a small-sample experiment may be non-significant and get abandoned. Significance markers visually merge "statistically detectable" and "practically important" into one signal, misleading readers into judging importance by significance.

Why it happens

Significance is determined jointly by effect size and sample size (a p-value is, at its core, "the probability of seeing this or more extreme data if the true effect were zero"), so larger samples make the same effect yield smaller p-values. This produces a counterintuitive consequence: once the sample is large enough, any tiny difference becomes "significant"—high-traffic A/B tests almost always produce starred results whose effect sizes may be commercially meaningless. The misreading source is treating stars as an "intensity" label for the effect (three stars means "more significant" and therefore "better"), when the star count only reflects p-value thresholds crossed (0.05, 0.01, 0.001) and has no direct relation to the difference's absolute or relative size. Visual design amplifies the misreading: stars are usually more prominent than effect-size numbers (placed above bars, heavier in weight), so attention lands on the significance signal first and the effect size becomes secondary information.

Where it stops holding

Significance markers are not worthless: in low-sample, high-noise settings (early experiments, small markets), significance acts as a gatekeeper against treating noise as signal—it answers "is this difference worth investigating further?" The problem is not the marker itself but making it the only and most prominent signal. The effect-size/significance tension is most extreme at very large samples (everything is significant) and inverted at small samples (important effects may be non-significant); mid-scale contexts (hundreds to thousands of observations) have the greatest information overlap between the two, and the misreading cost is comparatively smallest. For dashboards aimed at "should we act," effect size plus confidence interval almost always carries more information than stars.

Applying it

• Wherever significance is annotated, always show the effect size alongside it (the difference in absolute or percentage terms) so both are readable together.
• Use visual weight to guide the correct reading order: effect-size numbers inside the bar or at the line's end (primary visual), stars shrunk into a corner (auxiliary marker).
• On business-decision dashboards, replace stars with "minimum detectable effect" or confidence intervals, presenting "the range of the difference" directly.
• Verification: ask a reader to state the takeaway from a starred chart; if the retelling contains only "significant / not significant" and no effect-size number, the layout has made significance the primary signal.

Related

• Same group: U8.03.2 Uncorrected multiple comparisons inflate significance markers systematically · U8.03.3 There is no visual cliff between significant and non-significant · U8.03.4 Charts annotating significance must state the test used · U8.03.5 Showing effect sizes with intervals conveys more than significance labels
• Nearby: U8.01.1 Point estimates hide interval information · U8.02.2 Overlapping intervals between two groups do not imply a non-significant difference
• Search terms: statistical significance · effect size · p-value visualization