Monitoring during rollout must cover exceptions and performance, not only business metrics

What it is

If a canary watches only conversion, GMV, or daily actives, it will miss crashes, timeouts, mistaken submits, and unrecoverable states before the business curve moves. Monitoring must cover exceptions and performance, not only business metrics. Exceptions include uncaught errors, task breaks, permission failures, and corrupt writes. Performance includes the latency distribution, timeout rate, and resource saturation—not a pretty mean. Business metrics lag and can be masked by local success; interaction failure shows up first in errors and speed. If the gate reads only business, it stays green after harm has already started.

Why it happens

Business metrics are insensitive to local faults: if 1% of users cannot finish at all, site-wide conversion may not show it. Even when people fall back to an old path or give up, revenue may be temporarily held up by other entries. Error rate, crashes, and p95 latency pin directly to affected sessions and signal earlier. Performance degradation changes strategy—retry, mash, switch device—and those behaviors reach the business one beat late. Watching only business also rewards the wrong repair: operational subsidy pulls conversion back while timeouts stay. Canary windows are often shorter than the settling time of business metrics; gating an hourly ship on weekly actives is a timescale mismatch.

Studying it

List three metric groups per stage: exceptions (errors, crashes, task breaks), performance (latency percentiles, timeouts), business (conversion and the like). Draw red lines on exceptions and performance in advance; business is auxiliary. Contrast before/after and in/out of the canary on contemporaneous cohorts. SRE’s SLI/SLO idea can be borrowed for HCI evaluation: define “critical task completable” as a monitorable event, not only a funnel end. Qualitative tickets should enter the exception count by severity on the same dashboard. Do not declare the experience a pass when business metrics are non-significant.

Where it stops holding

Brand-new features have no business baseline, so exceptions and performance should be the main gate and business only descriptive. Some experience harm barely emits an error code (humiliating copy, an entry that cannot be found); monitoring must be paired with directed follow-up and must not claim the dashboard covers all experience. Alert fatigue makes on-call mute alarms; performance thresholds need calibration per stage. Privacy may forbid fine session logs; aggregated exceptions then stand in for replay. On offline or high-sync-delay clients, performance signals arrive late and the gate window must lengthen with them.

Applying it

• Default the release dashboard to three columns: exceptions, performance, business; bind the gate to the first two.
• Define start—success/fail/timeout events for critical tasks, rather than tagging only the funnel end.
• Draw red lines on percentiles and counts, not means; drill once with injected errors to see whether the gate lights.
• If business recovers and exceptions have not fallen, do not promote the stage.

Related

• Same group: Q5.09.1 Rollout share usually increases by stage, each with a continue-or-pause gate · Q5.09.2 Feature flags must toggle independently per user · Q5.09.3 Pilot users must know they are in a trial so they can interpret failures
• Adjacent: Q5.06 Canary release and pilots · Q6.12 Continuous tracking and alerting of metrics
• Search terms: exception and performance monitoring · canary telemetry · SLI