A post-hoc explanation is not the true mechanism

What it is

After a loan is denied, the interface generates “because income is unstable.” That sentence is a second piece of language, found after the result, to house it. What the model actually scored on may be postcode, channel, or a correlate never written into the copy. A post-hoc explanation is not the true mechanism: the first is a narration of an output that already exists; the second is the computation that produced it.

Narration can be useful, and it can be wholly off. Usefulness does not automatically equal fidelity. This entry is about that nonequivalence, not about whether one should explain a single output or a whole behaviour.

Why it happens

When an opaque model is asked to “speak human,” the usual move is to train a talking add-on: a linear surrogate, an attention map, a generative rationale. The add-on is optimised to look like a reason a person can read, not to walk the same computation. The causal chain the user reads can then be orthogonal to the internal decision.

People default to assuming the narration comes from the apparatus that did the work — ask a cook why the dish is salty and you expect the act of adding salt, not a voice-over invented afterwards. Print the voice-over next to the result and it is read as mechanism. Miller notes that explanation is a social selection: which factors get said is already an edit. A post-hoc add-on hands the edit to a second model, with no guarantee of fidelity.

Studying it

Run the original model on a batch, then the explanation add-on, and check two things: if the factors the add-on lists are perturbed, does the original output actually move; if factors the add-on never mentioned are perturbed, is the original output in fact more sensitive. Independent variables: add-on type (generative rationale / surrogate / attention map), whether users are told “this is an after-the-fact account.” Dependent variables: fidelity (does the output move with the emphasised factors), belief about the mechanism, actual success rate after people change inputs according to the explanation.

Do not substitute “felt clearly explained” for fidelity. Subjective clarity and mechanistic alignment are two quantities.

Where it stops holding

In inspectable rule systems (an explicit decision table, a formula that unfolds step by step), the printed path can be the mechanism, and this nonequivalence narrows. On creative tasks, generative rationales are often taken as copy, not as causes, so the harm is smaller. Harm concentrates where there is a right and a wrong and the user will revise materials according to the reason. This entry does not treat the split between process and outcome layers.

Applying it

• Mark a post-hoc account as such: “this is an account of the result, not a replay of the computation.”
• Give a factor list only when you can perturb: change this field and the output does move. If you cannot show that, do not write “because.”
• Do not wrap factors unrelated to the internal score in fluent causal sentences. Short and checkable beats story-like.
• Check: revise one input along the factor the UI emphasises and see whether the output moves. If users follow it and nothing changes, the explanation is lying; take that line out of the product.

Related

• Same group: L5.01.1 Explaining the process and explaining the outcome are different things · L5.01.3 The object of explanation is the decision, not the model
• Nearby: L5.02 Local and Global Explanations · L5.08 Counterfactual Explanations · L5.03 Trust Calibration
• Search terms: post-hoc explanation · explanation fidelity · rationalisation