Error rate responds to interface changes more sensitively than satisfaction, making it a good regression monitor

What it is

Under the same interface change, error rates move immediately and clearly while satisfaction scores drift slowly and noisily—subjective judgment has inertia that absorbs or delays behavioral change. That sensitivity gap makes error rate (paired with completion) the preferred behavioral metric for release regression monitoring.

Why it happens

The gap comes from measurement character: errors are discrete behavioral events directly shaped by interaction structure—one broken path shows up in logs the next day; satisfaction is a delayed, compositional attitude judgment smoothed by expectation and memory, needing accumulated use to move, with low scale resolution and large individual variance. In statistical terms, error rates have smaller sampling variance and faster effect onset; satisfaction needs bigger samples and longer windows to detect the same effect. Regression monitoring wants "change detectable in time," which is exactly the behavioral metric's home turf.

Studying it

Turning error rate into a regression monitor takes three pieces: a stable caliber (error definitions, recovery tiering, fixed task set), a baseline library (normal error-rate ranges per version), and alert rules (thresholds above baseline fluctuation). The sensitivity gap itself can be verified in side-by-side studies: record error and satisfaction changes for the same releases and compare detection lag and hit rate, quantifying the behavioral metric's advantage. Note that sensitivity brings false alarms too; a baseline fluctuation model (stratified by task and population) is the prerequisite for alert quality.

Where it stops holding

Sensitive is not comprehensive: error rate only covers "tried and failed" events—broken flows, rising burden, and satisfaction decay can all leave error rate untouched—so regression monitoring should be an error-first multi-metric combination, not a single metric. Low-frequency tasks yield thin error samples; stratify monitoring by task traffic and give sparse tasks wider confidence intervals or pooled windows.

Applying it

• Add an error-rate regression check to the release checklist: compare core-task error rates against the previous version and the rolling baseline; block or hotfix on breach.
• Tier the alert rules: wide thresholds for overall error rate, zero tolerance for unrecoverable errors.
• Monitor satisfaction and error rate on different clocks: watch satisfaction on quarterly windows for trends, never for per-release regression calls.

Related

• Same group: B5.10.1 Error rate correlates with effectiveness but must be reported separately; their remedies differ · B5.10.2 Errors must be counted separately as recoverable and unrecoverable; merged counts hide severe problems · B5.10.3 A zero-error interface may just have converted errors into abandonment
• Nearby: R2 Design Systems and Engineering Delivery · Q4 Measurement and Reliability
• Search terms: regression testing usability · error monitoring · sensitive metrics