Conversion rates are not comparable when funnel steps are redefined

What it is

Each funnel step is an operational definition: which event counts as arrival, which event counts as completion. If a release recodes “place order” from tapping submit to payment success, or drops an optional step from the funnel, the numerator and denominator are no longer the same object. The chart can still be labeled checkout conversion and still cannot say which version is better. Inconsistent step definitions are not sampling noise; they silently replace the measurand.

Why it happens

A conversion rate is a ratio of two counts. Tightening the numerator (a stricter success) lowers conversion with no behavior change; changing the denominator (who is counted as entering) moves the rate even if the rest of the flow is untouched. Inserting or deleting steps reallocates drop-off that used to live in one bucket, so one step looks worse and another looks better. Double-firing completion from client and server, or collapsing preview and submit, changes what the ratio means. Version comparisons that do not freeze the step contract confound product change with ruler change.

Studying it

Write the event, success rule, and exclusions for every step, and version-control that contract. Before a cross-release comparison, build a definition table: does the event still mean the same user action, is success intent or outcome, do optional steps still count. If the definition moved, either replay the old contract on the new data or break the series and declare a new baseline—do not splice the numbers. For a suspicious jump, inspect the release notes and the contract before narrating an experience change. Publish conversion with a contract hash or document version so later readers can tell whether two measurements were isomorphic.

Where it stops holding

Unchanged event names do not guarantee unchanged triggers: exposure versus button visibility, client submit versus server confirmation. In an experiment the two arms must share one contract; if the new flow has fewer steps, report both “aligned to the old steps” and “native to the new flow,” and pre-specify which one decides. Extremely sparse steps have unstable ratios even when definitions match, and should not referee a release.

Applying it

• Maintain a step contract: entry event, completion event, time window, exclusions. Treat a contract edit as a breaking change.
• Fill a definition table before any version comparison; if one step is misaligned, do not report conversion as up or down.
• After a flow change, keep a shadow metric on the old contract for at least one release cycle to separate ruler change from behavior change.
• Show the contract version next to the number; only points that share a contract may be joined as a trend.

Related

• Same group: Q3.12.1 Funnels locate drop-off, not causes · Q3.12.2 Retention curve shape outweighs a single-day number · Q3.12.3 Cohort analysis keeps new and returning users apart · Q3.12.5 Merging entry paths hides a path’s true conversion · Q3.12.6 Survivor bias overstates typical long-term experience · Q3.12.7 Rolling and classic retention are not interchangeable
• Adjacent: Q3.11 Event logging and instrumentation · Q3.06 Task success rate
• Search terms: funnel step definition · metric comparability · operationalization drift