Overly tight alert thresholds produce frequent false alarms that then get ignored

What it is

Set the alert line tight and it will fire almost every day. Fire often enough, and the person on call learns to delay opening, close in batches, or mute the channel. That is alert fatigue from tight thresholds. Once fatigue sets in, true anomalies travel the same ignored channel. Tightness is not caution. It turns alerts from exception signals into background noise.

Why it happens

Human response budget is finite. Every alert costs time to open, judge, and log; when most are later labeled fluctuation, the expected value of opening falls, and delay becomes a rational strategy. After an incident, systems often tighten the threshold “so we do not miss again,” false alarms rise, response falls further, and misses return in another form: not that the line was too wide to fire, but that it fired and nobody treated it as real. Experience metrics fatten this pattern, because they are noisier and more weekday-shaped than crashes, yet they are often given infrastructure-style minute-level hard thresholds. Fatigue also pollutes culture: a channel that has run empty too often loses political standing, and later even a reasonable alert cannot get into the meeting.

Studying it

For each alert rule, count daily fires, time-to-open, fraction labeled false, and whether confirmed incidents were handled in time. High false rate plus rising delay across weeks is fatigue forming. Replay incidents: after a tighten, did time-to-detect actually fall, or did only close-counts rise. A channel experiment also works: move low-value rules off the immediate channel and see whether time-to-open on high-value rules falls.

Where it stops holding

Life-safety and money failures can accept a higher false-alarm rate because misses cost more; even then, split the channel so they are not buried in experience noise. An overly wide threshold is the other error; anti-fatigue must not be an excuse to switch off a critical rule. Fatigue sometimes comes from unreadable alert copy rather than the threshold; rewriting copy may beat moving the line. Automated closing of “duplicate alerts” can also close a continuing anomaly from the same root cause.

Applying it

• Give each rule a daily cap that can still be investigated; if exceeded, widen or split first rather than asking people to close faster.
• After an incident, diagnose why it was missed; do not default to tightening the line one notch as the only fix.
• Keep only high-cost failures on the immediate channel; experience fluctuation goes to a daily digest, not a midnight page.
• Publish weekly false-alarm rate and time-to-open; rising delay is channel-health degradation, treated on par with a miss.

Related

• Same group: Q6.12.1 Continuous tracking must separate normal fluctuation from true anomalies · Q6.12.3 Metric definition changes need version labels, or historical trends will be misread · Q6.12.4 An alert only flags that an anomaly occurred; finding the cause still requires human investigation
• Adjacent: Q6.08 Relating experience and business metrics · Q3.11 Logs and event analytics
• Search terms: alert fatigue · false alarm · threshold tuning