Two overlapping confidence intervals don't automatically mean the difference between groups isn't real

What it is

On charts comparing two groups, a common reader inference is: "the error bars overlap, so the difference between the two groups is not significant." This inference does not hold. The degree of overlap between two 95% confidence intervals has no simple correspondence to whether the difference between the groups is significant—even with a substantial overlap, the difference may still be statistically significant. The reverse holds more reliably: non-overlapping intervals almost always imply a significant difference, but overlap does not imply non-significance. Judging significance by eyeballing overlap is one of the most common statistical misreadings in visualization.

Why it happens

The misreading stems from conflating two different comparisons: a single mean's confidence interval answers "where might this group's true value be," while the confidence interval for the difference between two groups answers "where might the difference between the two true values be." The width of the difference interval is not a simple combination of the two single-group intervals—it depends on both groups' variances and sample sizes, and paired versus independent designs compute it entirely differently. In the independent t-test scenario, two 95% CIs must shrink their overlap to roughly half (one bar's upper end crossing only the other's mean, not its interval boundary) before the difference approaches the significance boundary; in other words, with some overlap the difference may still be significant. The visualization presents only two single-group intervals, not the difference interval, leading readers to infer significance from the former's overlap—crossing a non-trivial mathematical relationship that eyeballing cannot substitute for.

Studying it

The classic paradigm for studying this misreading is a judgment task: present charts of two group means with their confidence intervals, have participants judge whether the difference is significant, and compute judgment accuracy against actual hypothesis-test results. The dependent variable is the incidence of the "overlap → not significant" inference. Variables include overlap amount (from no overlap to substantial overlap), sample-size annotation, and participants' statistical background. Findings consistently show that a substantial proportion of statistically trained readers also use the overlap heuristic, and misjudgment rates rise with overlap. Remediation directions include annotating the difference's confidence interval or p-value alongside the chart, and using dedicated difference plots that draw the quantity readers need to infer. A methodological caveat: in the lab, "is it significant" has a standard answer, while in real use readers usually need only "is this difference worth acting on"—the two judgments correlate only loosely.

Where it stops holding

The overlap heuristic's error directions are asymmetric: inferring significance from non-overlap is almost always correct (conservative, low error), while inferring non-significance from overlap is the high-risk misreading (a real effect gets dismissed as noise). In practice, the second error type usually costs more—missing a real effect is more common and more expensive than chasing a nonexistent one. For paired designs (the same users pre/post), the link between single-group intervals and difference significance weakens further, and eyeballing overlap is even less informative. Presenting the difference interval (the confidence interval of the between-group difference) is the fundamental fix, because it directly shows the quantity readers actually need to judge.

Applying it

• Supplement two-group comparison charts with the difference's confidence interval or hypothesis-test result rather than relying on readers eyeballing single-group intervals.
• Use a difference plot (the between-group difference and its interval) as a companion view that draws the inferred quantity directly.
• State in the chart's annotation that "interval overlap cannot be used to directly infer significance."
• Verification: take a two-group comparison chart and judge significance both by eyeballing overlap and by actual testing; a mismatch means the chart's design is guiding the misreading.

Related

• Same group: U8.02.1 Whether an error bar shows standard deviation, standard error, or confidence interval must be stated · U8.02.3 Interval width varies with sample size; sample size must accompany it · U8.02.4 Gradient bands create less certainty illusion than hard-edged intervals · U8.02.5 A band's visual width depends on the y-axis scale and cannot be compared across charts
• Nearby: U8.01.1 Point estimates hide interval information · U8.02.1 Whether an error bar shows standard deviation, standard error, or confidence interval must be stated
• Search terms: overlapping confidence intervals · significance judgment · difference plot