Guardrail metrics prevent one-sided optimization
Aliases: constraint metric · north star guardrail · floor metric
What it is
Guardrail metrics are constraints paired with a north star: the north star is allowed to rise, but named other outcomes must not cross a floor. Unsubscribe rate, task failure, support escalation, crashes, and complaints often sit here. A guardrail is not a second north star and does not compete over which number is larger. It answers only whether movement along that direction has stepped on an unacceptable edge. A single direction with no guardrails legalizes every lever that can lift the star.
Why it happens
The north star supplies a gradient; guardrails supply the feasible region. Optimization walks the gradient until it hits a constraint: more frequency lifts actives until unsubscribes and complaints stop it; fewer steps lift completion until risk-control failures stop it. Writing those edges as counts that must be watched is an advance admission that a single direction will overshoot. A guardrail works only if it can actually veto: even a clearly better north star must not fully ship if a guardrail is breached. If the guardrail is merely another chart to “also look at,” it yields in conflict, because rewards are already tied to the star. Choose the edges the north star is most likely to injure, not a friendly row of correlated extras.
Studying it
Back out candidates from historical full-ship incidents: during north-star rises, which counts worsened first and were later confirmed as harm. On the experiment platform, make guardrails a hard gate; record the fraction of launches stopped by a guardrail veto, and what would have happened if they had shipped (small holdbacks or later rollbacks). Compare net change on adjacent harm indicators under “north star only” versus “north star plus guardrails.” A guardrail that never fires may have too wide a threshold or may be watching the wrong edge.
Where it stops holding
Too many guardrails erase direction: when every column can veto, teams stop moving. Setting revenue, actives, and satisfaction all as guardrails hollows out the north star. Breach needs a pre-defined threshold and window, or “it looks close” becomes a judgment that can be lobbied at any time. Guardrails are not penalty columns either: they constrain overshoot; they do not require those counts to be optimized to their best. A count highly collinear with the north star cannot serve as a guardrail, because it will not alarm on its own when overshoot happens.
Applying it
- List two to four edges the north star is most likely to injure; give each a guardrail and a threshold that must not be crossed.
- Write the ship rule as a conjunction: north star met and no guardrail breached; missing either condition blocks full ship.
- Watch guardrails; do not give them an upward target, or they become a second star.
- Twice a year, audit the set against incidents: replace edges that never alarm, and add newly observed harm edges.