Uncorrectable inferences get switched off wholesale
Aliases: automation disuse · abandonment · trust generalisation
What it is
When an intent-inference system offers no means of correction, users do not respond to individual errors by "tolerating this one" — they switch the entire proactive feature off and fall back to manual operation. This is the disuse phenomenon from automation research: systems get disabled wholesale rather than amended locally. The user's arithmetic is plain — my own manual errors are mine to own, controllable; automatic errors I can neither pre-empt nor amend, so don't automate.
"Wholesale" is the operative word, and the cruel part: one uncorrectable error contaminates the whole feature category. One wrong auto-temperature adjustment, and what gets switched off is not that mistaken inference but the ninety-nine it got right. Users are under no obligation to distinguish "wrong this time" from "always wrong" — and when the system gives them no means to distinguish (no correction, no explanation), the blanket off-switch is rational loss-cutting.
Why it happens
Why does one error generalise to the whole feature? Trust research (Lee and See's synthesis on trust in automation) supplies the mechanism: trust is an overall dispositional judgement about a system, not a per-event ledger. People update it from a few salient samples — and uncorrectable errors are maximally salient (frustrating, disempowering, ruminated upon). One high-salience negative sample drags global trust far more than ten positives raise it, breaking the engineering intuition that "95% accuracy deserves trust".
A second layer is the learned-helplessness path. When correction is unavailable, users first still try to cope (working around it, pre-empting it); each attempt breaks against "can't change it"; after a few rounds, coping itself extinguishes, leaving only the master switch. Once helplessness sets in, even an improved feature struggles to win the user back — re-adoption demands fresh trust evidence from a user who has stopped watching.
A third layer is loss aversion asymmetry. Automation's gains are convenience (a few saved operations); its losses are episodes of lost control (the system acting against one's will). At equal objective magnitude, losses weigh heavier; uncorrectability tilts the loss side further still — a correctable error is friction, an uncorrectable one is an offence. Users will give up convenience to avoid offence, so the off-switch is a stable equilibrium, not an overreaction.
Studying it
- The classic evidence of disuse: automation-trust research documents the robust pattern that early errors cause lasting disuse — a system that errs before its reliability has been experienced gets switched off and rarely re-enabled, even after reliability genuinely improves.
- Calibration experiments: manipulate error correctability (undoable / reported-but-undoable / fully uncorrectable) and observe trust ratings, reliance behaviour, and feature-switch state; the uncorrectable condition shows sharply higher trust collapse and switch-off.
- Longitudinal smart-home deployments: trace the on/off trajectory of proactive features, identify the "one fault → permanent off" path per household, and measure re-enablement rates (usually near zero).
One methodological caution: switch-off is a late signal. Before turning a feature off, users typically pass through a long stretch of silent dissatisfaction (still using it, but lowered expectations, narrowed reliance); taking the feature switch as the only metric means detecting trust loss months after it实质 occurred. Longitudinal work needs intermediate variables (slow decline in reliance frequency, avoidance behaviour) to catch the disuse path early.
Where it stops holding
- Switch-off is not irrational — grant that first. Where the system is genuinely unreliable and correction is absent, disuse is the user's correct decision; reading switch-off rates simply as "users resisting novelty" misses the real signal — missing correctability.
- Correctable does not guarantee trust is retained. When correction exists but is costly (five steps, repeated interventions), disuse still follows; the bar is not "an entry point exists" but "an entry point at the consequence, at low cost".
- High-value features carry extra buffer. Features users depend on heavily (security, health) switch off more slowly at equal error rates — not because trust is stabler but because alternatives are worse; the buffer masks the problem until one big error settles the accumulated account in full.
Applying it
- Treat correctability as a release gate for proactive features, not a later patch: no inference feature ships without undo and feedback channels — a feature switched off after launch almost never gets a second chance.
- Intercept the user before the off-switch: monitor declining reliance frequency (the leading indicator of disuse), and when it slides, proactively ask "what has it been getting wrong lately", routing dissatisfaction into the correction flow.
- Keep a win-back channel for switched-off features: targeted notification after improvement ("the auto-climate you disabled: misjudgements down 80%") with a trial period; a generic "we're better now" recalls no one.
- How to check: track 30/90-day feature survival and the conditional probability of switch-off within 7 days of a misfire; the latter is the core health metric — it measures directly how lethal uncorrectable errors are to a feature.