Automation levels must match inference confidence

What it is

A system's judgement about user intent carries confidence — a probability, not a fact. The proactivity level must follow it: high confidence may execute automatically; low confidence falls back to suggesting or staying silent. This mapping principle is the operating manual for the proactivity spectrum: the same function should run at different levels under "I'm quite sure you want the light on" (person home, dark, walking toward it) and "I guess you sometimes want it on around now" (there is such a pattern in history).

The anti-pattern is treating level as a static attribute: the feature is defined as "execute automatically" and fires regardless of how certain each individual inference is. A frozen level forces every trigger to pay for the weakest inferences — and the low-confidence triggers contribute the large majority of errors.

Why it happens

Why must the match be dynamic? Decompose the cost: the expected error cost of one automatic execution is (1 − confidence) × consequence magnitude. Confidence is the multiplier: with trivial consequences, even low confidence is tolerable (wrongly turning off a light costs nothing); the heavier the consequence, the higher the confidence bar. Matching level to confidence means letting the moment of intervention track this product as it moves.

A second layer is the asymmetry of information value. When confidence is low, handing the decision back (downgrade to a suggestion) loses almost no information — one glance from the user corrects it. When confidence is high, execution spares the user even the glance. The downgrade mechanism converts the system's uncertainty into one cheap human confirmation — the best trade in implicit interaction.

Conversely, a static level discards the confidence information: knowing full well it is only sixty percent sure, the system still executes at full level, accumulating error cost as if every trigger were the worst case. What users feel is "it works sometimes and sometimes not", when the system could have said "not sure this time".

Studying it

• The decision-theoretic line: Horvitz and colleagues formalise "when to interrupt the user, when to act on one's own" as expected-utility comparisons — action value, inference confidence, and interruption cost enter one objective function, and the level choice falls out as its solution. The framework derives confidence thresholds rather than asserting numbers.
• Calibration research: work at the ML–HCI boundary examines calibration — whether a model's self-reported confidence matches its empirical accuracy. The matching principle presupposes confidence that is itself trustworthy; a systematically overconfident model executes triggers that should have been downgraded. Standard measurements are reliability diagrams and expected calibration error.
• Interaction experiments: compare a fixed-level version of an automatic feature against a confidence-adaptive one, with misfire rate, user corrections, trust scales, and perceived predictability as outcomes.

One methodological caution: evaluate matching on more than total error rate. Adaptive downgrading reshapes the error distribution — it trades "big errors from automatic execution" for "small frictions of being asked". The two costs must be tallied separately, or the experiment concludes, falsely, that matching makes things worse.

Where it stops holding

• The premise is usable, calibrated confidence. Where a model produces no meaningful confidence (or is badly overconfident), matching by confidence is matching by noise; the only sound choice is a conservative, fixed low level.
• Confidence is not the only dimension. At equal confidence, consequence magnitude and reversibility still independently set the safety bar — high confidence × irreversible consequence still must not run fully automatic. Confidence decides how far along the spectrum to slide; it does not waive the consequence dimension.
• Users cannot perceive a flickering level. A feature oscillating between suggesting and executing defeats stable expectations; matching needs smoothing and hysteresis, not per-trigger jitter.

Applying it

• Define a confidence threshold curve per proactive feature, not a single level: low-consequence actions may act at eighty percent confidence; heavy-consequence actions need near-certainty.
• Make the downgrade path concrete: where confidence falls short — silence / notification / suggestion with one tap — must be defined in advance, and shown to the user with a perceivable reason ("not sure — turn on the light?") rather than as silent non-response.
• Add hysteresis: after dropping from automatic to suggestion, require several consecutive correct judgements before promoting again, so the level does not oscillate.
• How to check: bucket past triggers by confidence and check that accuracy is monotonic across buckets (higher-confidence buckets genuinely err less); then measure the share of low-confidence triggers that still executed — a large share means the match is not working.

Related

• Same group: Z3.01.1 From notification through suggestion to automatic execution is a continuum · Z3.01.3 Levels should be user-settable
• Nearby: Z3.02 Boundaries of automatic execution · Z2.02 Uncertainty of inference
• Search terms: confidence-matched automation · calibration · decision-theoretic interruption · uncertainty-aware interaction