A year-over-year comparison is meaningless if last year's numbers were counted by different rules

What it is

Statistical scope is a metric's definition boundary: which events count, how duplicates are removed, which timestamp anchors the period, and what fields feed the numerator and denominator. Year-over-year and period-over-period comparisons presuppose that the baseline period measures the same thing as the current period. When the scope shifts, the ratio no longer reflects business change—it reflects a change in definition. This distortion produces no wrong numbers; every figure is correct, but the conclusion drawn from reading them together is wrong.

Why it happens

The distortion comes from the math of percentage change, which operates on values, not semantics: (current - baseline) / baseline assumes both operands share a measurement space. When scope changes—a new channel is included, deduplication moves from device to account, the attribution window expands from 7 to 28 days—the numerator and denominator each measure something different, the meaning of the ratio is broken, yet the calculation succeeds and the output looks normal. Humans are visual pattern recognizers: the growth curve on the dashboard keeps its shape, and no one goes looking for a definition change. This is precisely why the distortion is hard to catch—it leaves no visual trace: no NaN, no broken axis, no color anomaly. The remedy is to treat scope changes as first-class events: a scope change must be marked and visible on the timeline (a vertical reference line with annotation), and affected comparison ranges must explicitly read "scope changed, not comparable."

Studying it

Studying the impact of scope mismatch requires synthetic data with known ground truth: apply two scopes (e.g., different deduplication rules) to the same underlying event sequence, compute both sets of percentages, and quantify the spurious signal introduced by the scope change. Variables include the type of scope difference (definition expansion, attribution window, dedup rule) and the true magnitude of the underlying trend. In dashboard evaluation studies, one can measure users' notice rate for a "scope changed" annotation and the accuracy of their subsequent interpretation, testing whether the annotation design is sufficient to prevent wrong conclusions. A methodological caveat: in real data, scope changes often co-occur with genuine business changes—a natural confound—so synthetic data is the only clean route to causal isolation.

Where it stops holding

Scope mismatch and data-quality problems (late-arriving events, pipeline delay) are two different distortions: the former is a changed definition, the latter is data that has not yet arrived. The remedies differ—scope issues need annotations and documentation, delay issues need refresh indicators and backfill mechanisms. Scope consistency is necessary but not sufficient for comparability: seasonality, one-off events, and macro shocks can make two identically-defined periods incomparable. In exploratory analysis contexts, letting users freely change scope is a legitimate feature, but the change must be immediately visible and traceable.

Applying it

• Make every dashboard metric's definition reachable: hover or click to see the metric dictionary entry (numerator, denominator, dedup rule, attribution window, timestamp field).
• Mark scope-change dates with vertical lines along the timeline; comparison ranges spanning ±30 days around the change display a "scope not comparable" banner.
• Manage metric definitions centrally; scope changes go through approval and keep a version history in the dictionary entry.
• Verification: audit every percentage comparison on the dashboard, checking one by one whether numerator and denominator use identical definitions in both periods; any definition mismatch is a non-comparable instance.

Related

• Same group: U7.05.1 A single value means something only against a baseline · U7.05.2 Period-over-period suffers cyclic interference; year-over-year removes cycles but reacts late · U7.05.3 The baseline choice can flip the same metric into a rise or a fall · U7.05.4 With a tiny base, percentage change inflates into distortion
• Nearby: U10.02.1 The chosen time window decides the trend's direction · U10.04.4 Check whether uncertainty, missing proportion, and sample size are labeled
• Search terms: metric definition · statistical scope · comparability