Rolling retention and classic retention use different formulas and cannot be compared

What it is

Classic retention (N-day) asks whether a user was active on calendar day N after acquisition. Rolling retention (unbounded) asks whether they appeared on day N or any later day—whether they have not yet fully left by that age. Denominators can match; numerators do not. Rolling numbers are almost always higher. Comparing one formula to another team’s baseline, a competitor, or last quarter’s series attributes a formula gap to the product.

Why it happens

Classic retention is sensitive to which day someone returns, which fits products with a steady daily rhythm; a person who misses the target day and comes back the next is counted as gone. Rolling retention credits any return on or after the target day, closer to “not yet churned,” so it is monotone nonincreasing and kinder to sporadic use. Bracket or window retention, which counts activity in a neighborhood of the target day, sits between them. Teams reuse the label “seven-day retention” for whichever engine the dashboard happens to run. Switching the engine lifts or drops the whole curve, and even the shape’s meaning changes: a flattening rolling curve can just be late returns being counted forever.

Studying it

Lock the formula in the analysis plan: numerator event, calendar day versus rolling window, timezone, whether the target day itself is required. All comparison arms share that formula. External numbers require reconstructing the other party’s formula; if that fails, declare incomparability. Computing both classic and rolling on the same cohort is a diagnostic of how occasional users are treated, not two product results to rank. A formula change is a measurement change: break the trend line.

Where it stops holding

Products whose natural gap is much longer than N—quarterly reports, travel booking—make classic N-day rates look systematically poor; rolling or a business-cycle window is more apt, which still does not license comparing them to someone else’s classic number. Timezone, week-start, and the activity event (open, foreground session, critical task) widen the formula gap further. Rolling retention is unstable at the data cutoff: the newest cohorts have no “any later day” yet and will be undercounted until the window closes.

Applying it

• Register classic and rolling retention as separate entries in the metric dictionary; they may not share a name. Put the formula in the chart title.
• Before comparing with history or an external benchmark, check the numerator; refuse to place unmatched formulas side by side.
• When switching formulas, keep a shadow series on the old one through a full reporting cycle.
• Check: compute both seven-day numbers on one cohort; if the team cannot explain the gap, quote neither number externally.

Related

• Same group: Q3.12.1 Funnels locate drop-off, not causes · Q3.12.2 Retention curve shape outweighs a single-day number · Q3.12.3 Cohort analysis keeps new and returning users apart · Q3.12.4 Inconsistent step definitions make conversion incomparable · Q3.12.5 Merging entry paths hides a path’s true conversion · Q3.12.6 Survivor bias overstates typical long-term experience
• Adjacent: Q6.02 Goals, signals, and metrics · Q6.12 Continuous tracking and alerting
• Search terms: classic retention · rolling retention · unbounded retention