Rollback criteria must be explicit
Aliases: kill criteria · canary rollback rule
What it is
A limited release without written rollback criteria (kill criteria) turns failure into an argument in a meeting: how many complaints count as serious, whether one data-loss event stops the train, how far error rate may rise above baseline. Rollback criteria are observable, decidable, scope-bound stop rules; when they fire, users return to the old experience or the new path closes. They are not the engineering ability to roll back—without criteria that ability is a locked door. Criteria must freeze before results are seen, or the team will redefine “still fine” after people are already hurt.
Why it happens
Post-release information is muddy: novelty effects, lagged tickets, internal staff explaining things away. Without thresholds, responsibility diffuses and everyone can narrate the present as “watch a bit longer.” Harm continues during the argument. Criteria set in advance turn judgment from preference into a rule: hit, then stop; miss, then keep collecting. They also force the team to imagine failure shapes—crashes, mistaken submits, blocked critical tasks, complaint spikes—and therefore to add monitoring. Rollback itself has costs (new habits, unfinished migrations), so criteria should separate “stop now” from “do not enlarge this stage,” rather than treating every anomaly as a pull of the brake.
Studying it
Treat rollback criteria as part of the protocol: metrics, baselines, time windows, and the decision-maker. Tabletop a failure and see whether current monitoring would fire inside the window. After action, report whether criteria fired, whether they fired late, and whether they were rewritten. Rewritten criteria are no longer a prior commitment. Qualitative severe cases need an escalation path into rollback, not aggregation-only rules. This is kin to a trial stopping rule: protecting participants outranks protecting the hypothesis.
Where it stops holding
Some failures have no clean rollback: data already written to an external system, content already sent, a filing already made with a regulator. Criteria then need compensation and isolation, not only “cut back to the old build.” High-lag metrics (weekly actives, NPS) cannot be instant rollback grounds. Criteria that are too narrow treat real harm as noise; too wide treat ordinary jitter as disaster and burn trust in the method. Multi-person sign-off slows firing; one person alone may false-stop—write the on-call duty, not “we’ll discuss.”
Applying it
- Before release, freeze three lines: roll back now, stop enlarging, keep watching; each with a metric, a window, and an owner.
- Tabletop a failure to confirm monitoring lights inside the window; failed drills block release.
- Give severe cases their own escalation; do not require statistical significance first.
- Fire, then explain. Do not reconvene to redefine a threshold that has already been hit.