Metric definition changes need version labels, or historical trends will be misread

What it is

Once the counting rule for completion, actives, or satisfaction changes denominator, exclusions, event names, or survey items, the new number is not the same series as the old. Metric definition versioning requires each change to carry a version, an effective time, and a diff, and to break or parallel the chart rather than append the new rule to the old line. Without labels, an instrumentation fix looks like an experience surge, and excluding bots looks like an activity collapse.

Why it happens

A line chart is read by default as continuous observation of one generating process. Definition is part of that process: changing it changes the ruler. Neither the eye nor automatic alerts know the ruler changed, so they book ruler difference as product difference. Changes often happen on the engineering side—event rename, dedupe logic, sampling frame—and experience teams that only consume reports tell the wrong story without knowing, even shipping or alerting on it. Versioning splits “the number has the same name” from “the number is still the same definition.” The name may continue; the definition must have generations. Without generations, history is unauditable: nobody can say whether 72% three months ago and 72% today are the same thing.

Studying it

Audit whether every series on the live dashboard has a complete definition-change log, and check several inflections for a contemporaneous definition event. Run a splice experiment: join a known definition change on one axis and ask colleagues who were not told to explain the inflection; count how many narrate it as a product change. Where dual computation is possible, run old and new definitions in parallel and estimate the break size as a later chart note. Align alert rules the same way: did the definition-change day false-fire or miss.

Where it stops holding

Spelling fixes or display-only renames need not bump the major version, but should still log a patch so search does not rot. When the old definition cannot be replayed (raw events gone), the break must be drawn as a discontinuity; interpolation must not fake continuity. Changing definition too often makes versions themselves unreadable; collect changes into planned windows rather than rewriting the definition on every commit. Versioning does not forbid improving a definition. It forbids pretending, after the improvement, that history is still directly comparable.

Applying it

• Show current version and effective date in the legend of every default metric; draw a vertical mark on the change day.
• A definition change must run the old definition in an overlap window; do not delete the old series before overlap ends.
• External or upward reports that cross a change day must include the sentence “the ruler changed here.”
• Bind alert rules to definition version; re-estimate thresholds on a bump; do not keep the old line.

Related

• Same group: Q6.12.1 Continuous tracking must separate normal fluctuation from true anomalies · Q6.12.2 Overly tight alert thresholds produce frequent false alarms that then get ignored · Q6.12.4 An alert only flags that an anomaly occurred; finding the cause still requires human investigation
• Adjacent: Q6.02 Goals–Signals–Metrics · Q6.11 Benchmarks and competitive comparison
• Search terms: metric versioning · definition change · broken time series