High-stakes actions should not fire on a single inference
Aliases: confidence gating · mixed-initiative · action confirmation
What it is
Inference carries an error rate; risk = error rate × consequences. Low-stakes actions (a light, some music) can fire on a single inference — the error is negligible. High-stakes actions — unlocking, paying, sending outward, anything irreversible — cannot bear even one error and must not fire on a single probabilistic judgement. They require escalation: corroboration across sources, repeated triggering within a time window, or immediate user confirmation — at least one of the three.
The criterion is not how "important" the action feels but its reversibility after error and its outward reach: a wrongly lit lamp can be switched off; a wrongly opened door cannot be recalled; a wrongly sent message is already gone.
Why it happens
The magnitudes decide that this is not a matter of taste: at best, activity recognition errs one to two times in ten — a single inference errs at the percentage level, not the fourth decimal. In low-stakes settings daily life absorbs those errors (nobody notices one extra lamp-on); in high-stakes settings it is a fixed probability of disaster — a daily high-stakes automation at a two-in-ten error rate fails within the week with near-certainty.
Each escalation mechanism, with its own logic and price:
- Multi-source corroboration: fire only when two independent modalities agree. Presupposes non-overlapping failure conditions (otherwise one interferer fools both); the price is more misses (both must see).
- Repeated triggering: fire only after several consistent inferences in a window. Works on uncorrelated errors; useless against systematic bias (sensor drift, two family members of similar build) — repeated sampling cannot wash out errors from a shared source.
- Immediate confirmation: return the final decision to the user. Safest, but hands back some or all of the automation's convenience; at high frequency, confirmation fatigue sets in and users tap "yes" mechanically — the check becomes decoration.
Studying it
- Mixed-initiative principles: Horvitz's 1999 principles for mixed-initiative interaction supply the decision framework — the value of automation weighed against waiting cost and error cost jointly determines whether and when to hand the decision back. Firing a high-stakes action on a single inference is precisely a violation of this trade.
- Decision-theoretic formulation: expected cost of a false action = P(misinference) × cost. The cost-weighting logic of signal detection theory applies directly: the firing threshold belongs at the point where expected cost of a misfire equals expected cost of a miss — for high-stakes actions that point sits far toward the conservative end.
- Confirmation fatigue: a classic finding from security research — as confirmation frequency rises, user compliance rises while reading rate falls (generalised as alarm fatigue, well documented in clinical alarm management). Use it to evaluate whether "confirm every time" actually protects anything.
Methodological caution: evaluate escalation on both error directions (misfire and miss rates) plus user burden (confirmations per day, response time) — a scheme optimised on one will degrade on the other two.
Where it stops holding
- High-frequency actions can tip the trade the other way. A "leaving home" mode that demands confirmation every single departure accumulates more confirmation cost than the occasional error would cost — the right fix is not more confirmation but lower stakes (make it reversible) or better inference.
- Emergency scenarios invert the design. For functions where missing the event is catastrophic — fall detection — prefer false alarms: lower the threshold, ask actively, escalate to help on timeout. The escalation logic runs backwards.
- Repeated triggering cannot fix systematic error. It suppresses uncorrelated noise; recognition confusions (one family member read as another) are systematic and repeat identically a hundred times.
Applying it
- Grade the automation inventory on two axes: reversible/irreversible and inward/outward. Anything in the irreversible or outward quadrant is forced onto an escalation path.
- For each high-stakes action, fix the escalation parameters: how many corroborating sources, how many consistent inferences within what window, and the timeout default for confirmations (timeout must not default to "execute").
- Define an error budget per tier (misfires allowed per thousand) and audit against it after launch.
- How to check: inject misinference — replace the recognition engine's output with wrong labels and watch whether the system still fires the high-stakes action. Any path that fires on a single injected point is a defect.