A spike in reports is itself an alarm

What it is

One report means someone failed. A sudden rise in the same kind per unit time means a path broke for many people at once—a release, a dependency, a config, an attack. The elevated rate is itself an alarm; do not wait for tickets to pile up or for someone to happen to look at a dashboard. This entry is about treating rate as a signal. It is not about what a single payload contains.

Why it happens

Interaction failures have a baseline in quiet periods: one region’s network, one old device, one rarely taken path. A jump in that baseline points faster than any single stack to “this is not what it was ten minutes ago.” Complaints lag; automatic report rate almost does not. Alarms must convert “count” into “relative to our own baseline”: same version, same hour versus last week, not one global absolute threshold—rising daily actives lift absolutes too. An unalarmed spike is handled as an incident after users have already left for another product. False alarms train the on-call to ignore, so a real spike is clicked through—the same habituation that kills confirmations.

Studying it

Replay historic report series: run different detectors (absolute threshold, relative baseline, per version) on known incident windows and quiet windows.

Independent variables: detector, window length, whether sliced by version or region. Dependent variables: detection delay on known incidents, false alarms on quiet windows, whether on-call starts ignoring a class of alert.

Do not use “we felt this one should fire” as the label. Use post-confirmed incident windows.

Where it stops holding

A brief spike at release time may be clients upgrading together; align with version so every ship is not an incident. Wrong client clocks create fake spikes; server receive time is more reliable. One user retrying the same failure inflates rate; dedupe per user before comparing to baseline, or one person’s loop equals a crowd. Quiet hours usually do not apply to interaction failures: people still fail at night.

Applying it

• Build a baseline per critical error kind; alarm on relative change, with version and path slices attached.
• Fold one user’s short-interval duplicates before counting, so retry policy does not pollute rate.
• An alarm must click through to sample paths, not only dump an integer.
• Verify with a real or drilled fault: time from rate climb to a human owning it. If tickets were how you knew, rate is not yet a signal.

Related

• Within the group: H3.14.1 Client errors must report themselves · H3.14.2 Reports need the path, not only the stack · H3.14.4 User content in reports has to be redacted
• Adjacent: H5.08 Alarm fatigue · Q3.08 Error rate and help-seeking rate · H3.10 Retry strategy
• Search terms: error spike · telemetry alert · baseline