Users often cannot tell why the system did that

What it is

When intent inference fails — the automation does something the user did not want — the first obstacle is not "how do I change it" but "why did it do that": the sensor signals behind the inference are invisible, the judgement is not shown, and the user is left holding an action with no reason attached. This state is the attribution gap: behaviour without basis. The smart speaker's false wake is the prototype scenario: the speaker suddenly answers, the user never said the wake word, and will never know what it "heard".

The attribution gap amplifies inference failures: the same error, with a visible cause ("I bumped the sensor"), can be filed away and prevented; without one it gets chewed over — no way to know when it will recur, no way to defend, and eventually it settles into diffuse unease about the whole system.

Why it happens

Attribution needs material: observable inputs, traceable judgement, rules to check against. Intent-inference systems supply none of the three — structurally, not by oversight:

• Inputs are invisible. The inference consumes sensor streams (acoustic features, location, heart rate, power curves) that the user cannot perceive, so "did it mishear?" cannot be tested.
• The process is opaque. A probabilistic judgement has no rule to consult; the model holds no human-readable "because you usually… at this time". Even when a reason is emitted, it is often a post-hoc template unrelated to what actually triggered.
• Expectations cannot form. Users cannot learn the boundary from observing failures — "when will it misread" has no answer in front of a black box.

Norman's writing on smart machines names the second layer: the "smarter" the behaviour, the more users tend to over-attribute — granting it understanding, intentions, even temperament. Misfires get explained as "it gets me" or "it's broken"; both folk theories sit far from the real cause, and both impede the right response.

Studying it

• Phenomenology of unwanted activations: log-driven false-wake statistics paired with user interviews — users' guesses at "why it woke" scatter widely and are mostly wrong; the most direct evidence of the gap.
• In-home interviews about automations: have users explain their own automations' trigger conditions and compare against actual configuration; studies find users largely hold confident but wrong trigger theories, which steer wrong responses during faults.
• Attribution experiments: present the same error under different explanation conditions (none / template / specific signal) and compare trust repair, recurrence expectation, and subsequent behaviour.

One methodological caution: users will supply a reason whether or not one is given — "it probably heard the TV". Measuring the gap cannot stop at "do you know why" (everyone has an answer); check answers against real trigger logs, and weigh confidence — a confident wrong theory does more damage than "I don't know".

Where it stops holding

• The gap surfaces only at failure. When the judgement is right, no reason is needed and the gap is harmless — which is why products overlook it; evaluation must include failure scenarios, which happy-path testing never reaches.
• Not every explanation helps. A faithful signal-level explanation ("detected a sustained 43 Hz tone") is just jargon to most users; explanation granularity must match the user's model — a trade-off treated where explanation granularity belongs, not repeated here.
• The gap compounds with device count. With one device, users can still reason by elimination (only it ever speaks); with a dozen proactive devices, even "who did that" fails — attribution breaks before localisation. Multi-device attribution is a different order of problem.

Applying it

• Every executed intent-inference action carries a reason tag: where the action surfaces (notification, state change), one line of what the judgement was based on — "you left home and were inactive for 20 minutes, so the air conditioning was switched off" — never boilerplate, always the actual basis.
• Make the tag expandable: a plain-language layer, and beneath it a checkable signal fact (time, sensor, threshold) for users who want depth.
• Keep a post-mortem entry for misfires: after an error, retrieve "what was the signal, how confident was the judgement", turning each failure into learnable boundary knowledge.
• How to check: re-interview within 24 hours of a misfire and score explanations of "why did the system do that" against trigger logs; A/B the reason tag (present / absent) and compare post-misfire disable rates and speed of trust repair.

Related

• Same group: Z3.03.2 Correction entry points belong where the outcome occurs · Z3.03.3 Uncorrectable inferences get switched off wholesale
• Nearby: Z2.07 Making context judgements explainable and visible · Z7.01 Making system behaviour explainable
• Search terms: attribution gap · unwanted activations · explainable agent behaviour · smart speaker misactivation