A company-wide average can hide two subgroups moving in completely opposite directions

What it is

Collapsing data to an aggregate level (company-wide averages, national totals, site-wide conversion) makes subgroup differences vanish from view—not just averaging away individual variation but, in the extreme case, reversing the direction of every subgroup's conclusion. Simpson's paradox is that extreme form: two segments each show treatment A is better, yet the combined totals show B is better (because the segment sizes differ). Even in ordinary non-paradox cases, aggregation flattens "some subgroups rising, others falling" into "the whole is flat," and the "stability" readers infer from the aggregate actually conceals structural change in progress.

Why it happens

Aggregation hides differences through the structural folding of weighted averages: the aggregate is a weighted mean of subgroup values, with subgroup sizes as weights. When the size ratio between subgroups A and B shifts over time (channel A grows faster than B), the aggregate drifts even if each subgroup's internal metric is unchanged—rising overall conversion may come entirely from "the high-converting channel's share grew," with no improvement in any channel itself. Misreading aggregate drift as subgroup behavior is the ecological fallacy: inferring individual-level associations from group-level associations. Aggregation's necessity is equally real: decisions and reporting need concise totals, and everything cannot be expressed as detail. The issue is therefore not "whether to aggregate" but "whether an entry point for checking stratification survives the aggregation"—healthy analytical practice makes the aggregate the default view with stratification one click away, so an anomalous aggregate can be immediately drilled to confirm whether it is universal or driven by a particular subgroup.

Studying it

The standard method for studying aggregation effects is stratified comparison analysis: compute the target statistic on the same data at the aggregate level and at each subgroup level, testing direction consistency and magnitude differences. Simpson's paradox detection is the special case of checking whether subgroup directions oppose the aggregate direction. In visualization research, one can measure readers' accuracy in judging "which subgroup drove the change" when shown only the aggregate versus aggregate-plus-stratified views, quantifying stratified views' contribution to correct attribution. A methodological caveat: the subgrouping scheme itself is an analytical choice (extending into the same group's grouping effects), so stratified analysis should try multiple partitions (by channel, region, user segment) to avoid "happily choosing the partition that hides the paradox."

Where it stops holding

The cost of stratified display grows with subgroup count: 3 subgroups stratify readably, 30 do not—stratified checking for high-dimensional data needs assisted ordering (rank subgroups by their contribution to the aggregate) rather than full display. Another form of aggregation's masking is temporal: monthly aggregation masks within-month daily patterns (weekend effects), reminding us that stratified checks should run along multiple dimensions (group and time). Small subgroups are statistically unstable (small-sample noise), so subgroup differences seen after drilling may be noise rather than real structure—stratified interpretation must be paired with checking subgroup sample sizes.

Applying it

• Give aggregate charts a default stratification entry: click an aggregate value to expand subgroup decomposition, or provide a small companion stratified chart.
• When reporting aggregate changes, attach one line of attribution: "the change is driven mainly by subgroup X / contributed by all subgroups."
• For cross-period comparisons, check whether subgroup size ratios changed; when they shift beyond a threshold, report the aggregate change decomposed into "composition effect" and "behavior effect."
• Verification: run a stratified attribution on the most recent "overall metric change" and compare aggregate vs subgroup directions; opposing directions are an instance of aggregation masking.

Related

• Same group: U8.06.2 The choice of grouping can flip a conclusion's direction · U8.06.3 A drill-down path to raw granularity must be available
• Nearby: U8.06.2 The choice of grouping can flip a conclusion's direction · U7.06.1 The default time range determines the conclusion most users will see
• Search terms: simpson's paradox · ecological fallacy · aggregation bias