Crossing a threshold once and crossing it continuously are different problems needing different responses

What it is

Crossing the line once and crossing it continuously are different species of event: a single-point breach may be a spike (network jitter, one batch peak), while a sustained breach points at a real state change (service degradation, demand collapse). Their responses should be tiered — single points get logged for observation, sustained breaches escalate to alerts; a uniform policy produces both misses (real problems buried under spike alerts) and fatigue (users desensitised by frequent spike alerts).

Why it happens

The tiering follows from the signal's noise structure: instantaneous values carry heavy noise, and duration is a natural denoising filter — the condition of N consecutive periods over the line filters most random spikes while preserving genuine degradation, which usually persists. The engineering form is the sustained window: its core parameters are window length and sampling granularity, and together they set the alarm's sensitivity versus lag — longer windows mean fewer false alerts but later discovery, a trade that must follow the metric's tolerable response time (payment-path windows in seconds, daily-report windows in hours). Tiered response has a mirror too: recovery should also be windowed — no instant flip back to green when the line is re-crossed in the good direction, or the state oscillates at the boundary.

Where it stops holding

Tiering is not the more conservative the better: for should-never-happen hard errors (core service unreachable), alerting on the single point is correct — the noise assumption itself fails for such metrics, and applying a denoising window only adds discovery delay. Sustained-window parameters also need re-review as business rhythm changes (campaigns, new channels): the original window may no longer match the noise structure. Tiered alerting must render its tiers visibly — an "observing (1 of 5 periods)" state should be seen by users; if single-point breaches pass silently, users lose spike visibility and with it the ability to judge whether spikes are getting denser.

Applying it

• Configure a sustained condition for every alert rule (N consecutive periods over the line before alerting), N set by response tolerance.
• Separate two tiers on the alert panel — "observing" and "alerted" — with the observing tier visible but not pushed.
• Verification: replay a week of history and compare alert counts and true-problem hit rates before and after tiering; a big drop in alerts with an undamaged hit rate means the tuning worked.

Related

• Same group: U7.04.1 Thresholds convert a continuous metric into two discrete states · U7.04.2 The threshold's rationale must be visible and adjustable · U7.04.4 Too many alerts train users to ignore alerts · U7.04.5 Alerts must point to the executable next step, not just report the value
• Nearby: U7.09.1 Jumping data interferes with reading · U7.09.2 Update frequency must match decision frequency
• Search terms: alert for duration · flapping alert · alert severity levels