Users must be able to answer why the system did that

What it is

When an automated system acts on the user's behalf, the user must be able to answer one question: why did it do that. The property that supports this answer is intelligibility. Bellotti and Edwards set it alongside accountability as a baseline requirement for context-aware systems in 2001: the system must let users see what it senses and on what basis it acts, and must remain answerable after the fact.

This is not a "friendly UI" concern. In ambient computing most actions happen outside attention; what the user sees is the outcome — the light came on by itself, the door locked itself, the heating went up — not the process. Intelligibility requires the system to supply the "on what basis" afterwards; otherwise the user faces a black box whose behaviour cannot be predicted.

It also differs from documentation. A manual answers "how do I use this"; intelligibility answers "why did it just do that" — about a specific, already-occurred behaviour, not an abstract feature.

Why it happens

Use of an automated system runs on one causal chain: predict the next action → decide whether to rely on it → localise and correct when it errs. All three links consume the same input — the answer to "why did it do that".

Prediction runs on a mental model, and a mental model of an automation cannot be read off the device's appearance: actions are decided backstage by sensor conditions and rules. Explanations are the only building material for the model. Without them the model stalls at whatever the user imagined at installation time, and drifts further from reality as the system evolves.

Reliance is a trust-calibration problem. Users need to know under which conditions the system is dependable and under which it is not; only by being told the grounds of a judgement can they draw the boundary of "trust it" and exclude high-stakes situations from automation. An opaque system earns not rational trust but gambler's reliance — and collapses accordingly.

There is one further cost. A system whose behaviour has no causal answer also has no accountability: when something goes wrong, nobody can say whether the inference, the configuration, or the hardware was at fault, and responsibility is left hanging. Diagnosis is that problem's own territory, but its entrance is the explanation described here.

Studying it

The empirical base is the intelligibility research line:

• Information-demand surveys: Lim and Dey's assessment of what users actually want to ask when facing a context-aware system's behaviour, yielding a typology of explanation needs — why it acted, why it did not, what would happen under different conditions.
• Question-type toolkits: follow-on work organised explanation questions into a toolkit — why, why not, what-if, how-to — for designers to pick from per application.
• Controlled experiments: Lim, Dey and Avrahami compared versions with and without why / why-not explanations, measuring how accurately users could predict behaviour, and how trust and reliance calibrated. This is the paradigm that directly tests this entry.

Conventions for variables: independent variables are explanation type (why / why not / what-if) and presentation timing (at the behaviour / on request); dependent variables are prediction accuracy, trust scales, and reliance behaviour (whether users delegate the task). One methodological caution: lab participants have no real stake in the system, so absolute trust-scale values mean little; recent work measures reliance changes in deployments with real consequences or in the wild.

Where it stops holding

• Explanation needs rise with autonomy and consequence. A low-risk daily schedule needs almost none; autonomous actions with spillover (money, privacy, other people) make explanation a hard requirement. Explaining every action uniformly drowns the benefit in cost.
• Explanations carry an interruption cost. Explaining at the moment of behaviour forces an interruption; explaining only on request may be too late. Timing is itself a design variable with no free option.
• Intelligibility does not remove uncertainty. For probabilistic systems an explanation can only say which signals were used, not that the judgement was right — treating an explanation as proof of correctness is a mistake on both sides of the design.

Applying it

• Keep an after-the-fact causal record for every executed automation: which rule fired, which condition matched, the key sensor values at the time. No push required — but the question must be answerable.
• Organise explanations around the user's questions — why it acted, why it did not, what would change under different conditions — not around the system's internal structure.
• Put the entry to the causal record where the behaviour surfaces: one step from "why did the light go off by itself".
• How to check: sample recent system actions at random, ask users "why did it do that", and score first-answer accuracy; compare accuracy and trust before and after the explanation entry exists. If accuracy does not move, the explanations are decoration.

Related

• Same group: Z7.01.2 Unexplained automation gets attributed to malfunction · Z7.01.3 Explanations must point to causes the user can change
• Nearby: Z7.02 Fault diagnosis · Z2.07 Making context judgements visible and explainable
• Search terms: intelligibility · accountability · explainable intelligent systems · why not explanations