A p-value of 0.049 and 0.051 mean almost the same thing, yet one gets a star and the other doesn't

What it is

The difference between p=0.049 and p=0.051 in the strength of statistical evidence is nearly zero—yet chart visual language presents them as two different categories: one starred, one not; one solid-colored, one grayed. The significance threshold is a human convention (Fisher chose 0.05 for table-lookup convenience, not because it has mathematical significance), and nothing changes in substance the instant a p-value crosses it. When a visualization uses starkly different visual encodings on either side of the threshold, it implies a cliff that does not exist, and readers treat p=0.049 as "true" while treating p=0.051 as "nothing"—even though the actual credibility gap is far smaller than the visual presentation implies.

Why it happens

The cliff illusion stems from categorical encoding distorting a continuous quantity: the p-value is continuous, the significance judgment slices it at 0.05, and the visual encoding (star/no star, colored/gray) further reinforces "these are two categories." The cost of categorization in visual cognition is the boundary effect: human perception is highly sensitive to differences across category boundaries (colors on either side of a blue/green boundary are judged "different colors"), while large differences within a category are comparatively ignored (two different shades of green are judged "both green"). Applied to significance markers: p=0.049 vs 0.051 (across the boundary, tiny difference) is perceived as fundamentally different conclusions, while p=0.001 vs 0.049 (same side, large difference) is perceived as the same conclusion. Comparisons with vastly different evidence strength are visually equated. Moreover, the threshold's conventional nature means different fields, journals, and companies use different thresholds (0.05, 0.01, 0.1), yet the visual cliff on the chart looks equally absolute—the threshold's history and subjectivity are completely invisible in the visual form.

Where it stops holding

This does not mean significance judgment is worthless: in binary-decision contexts (ship or not, invest or not), the threshold is part of the decision rule, and a binary visual marker aligns with the decision structure. The problem is treating a decision aid as a measure of the evidence itself. "Borderline" results near the threshold should not be strongly interpreted in either direction—p=0.049 does not prove the effect exists, p=0.051 does not prove it absent; both express "insufficient evidence to conclude." For dashboard design, expressing a continuous evidence strength with a visual cliff wastes the chart's information capacity: the same chart can present continuous p-values or confidence intervals while marking the decision threshold more lightly (e.g., a secondary-color reference line).

Applying it

• Where possible, display the continuous p-value or confidence interval directly rather than only the binary post-threshold marker.
• When binary markers are required, encode distance from the threshold with visual weight (color intensity, marker size), avoiding the impression of two entirely different categories on either side.
• State the threshold's conventional origin in the caption ("per team standard α=0.05") so readers know it is a convention, not a natural boundary.
• Verification: find all comparison pairs on the chart treated differently solely by threshold proximity (e.g., p=0.048 vs p=0.052) and check whether their visual difference is proportionate to the actual gap in evidence strength.

Related

• Same group: U8.03.1 Significance markers do not convey the size of the effect · U8.03.2 Uncorrected multiple comparisons inflate significance markers systematically · U8.03.4 Charts annotating significance must state the test used · U8.03.5 Showing effect sizes with intervals conveys more than significance labels
• Nearby: U8.01.3 Omitting uncertainty claims certainty · U8.03.1 Significance markers do not convey the size of the effect
• Search terms: p-value threshold · dichotomization · boundary effect