An alert only flags that an anomaly occurred; finding the cause still requires human investigation

What it is

An alert answers “this count left the ordinary.” It does not answer “which ship, which event, which user class, which external dependency.” Alert is not diagnosis. A completion-rate drop may be the payment rail, a misplaced button in a new version, a rewritten definition, or a competitor promotion that siphoned high-intent users. Writing the cause into the alert title, or triggering rollback automatically, treats detection and attribution as one step.

Why it happens

Detection rules watch a marginal distribution or a deviation from baseline; that is only enough to say “unlike ordinary.” Causes live in the conditional distribution: once sliced by version, entry, device, region, event funnel, the deviation concentrates on one layer. Slicing needs a person to choose which dimensions, which logs, which still-awake humans to ask, because the combination space is larger than any pre-loaded explainer. Automation can prepare candidate slices; it cannot replace a judgment about mechanism: the same “payment failures up” slice may be an expired certificate or copy that made people abandon. Treating the alert as diagnosis too early also trains the system backwards: on-call staff take the alert copy as the conclusion and stop asking, so when the true cause sits outside the copy, mishandling repeats.

Studying it

Code closed anomaly tickets: alert copy, first claimed cause, final root cause, and whether they match. The mismatch rate is the cost of treating alert as diagnosis. Compare a flow that must finish a slice checklist before close with a flow that may close on the alert copy, on mishandling rate and time. Evaluate automatic attribution suggestions as suggestions rather than conclusions, and see whether people still verify. Count investigation time into channel design: if the diagnostic window is longer than the incident can bear, add a separate immediately executable safety action (circuit break) and state that it is not attribution.

Where it stops holding

A very narrow rule can approach diagnosis—“the certificate expires in two hours” is itself the cause; experience counts are rarely that narrow. Automatic rollback can be used when false alarms are reversible and harm is asymmetric; rollback is still not diagnosis, and the cause must be found afterwards or the same alert will recur on a cycle. Human investigation is not the same as an expert arriving: a pre-written slice list that on-call staff cut by is already structured human judgment. Alerting with no on-call capacity only piles “undiagnosed” into open tickets.

Applying it

• Alert copy names the metric, the direction of deviation, and the window, not a cause sentence; leave the cause field empty to fill.
• Before close, finish the pre-defined slices (version, entry, device, funnel) and one mechanism sentence; otherwise it is not resolved.
• If automatic rollback is on, the ticket state is “bleeding stopped, diagnosis pending”; it must not close.
• In review, separate detection quality from diagnosis quality; “the alert fired” does not prove the cause is known.

Related

• Same group: Q6.12.1 Continuous tracking must separate normal fluctuation from true anomalies · Q6.12.2 Overly tight alert thresholds produce frequent false alarms that then get ignored · Q6.12.3 Metric definition changes need version labels, or historical trends will be misread
• Adjacent: Q6.08 Relating experience and business metrics · Q3.11 Logs and event analytics
• Search terms: alert is not diagnosis · root cause · anomaly triage