The object of explanation is the decision, not the model

What it is

A claim is denied and the applicant asks why. What they need is not how many layers the net has, or which attention it uses, but what this particular decision turned on. Pasting a model card, an architecture diagram, or a note about training data answers a different thing. The object of explanation is the decision, not the model: the user has to be able to take a next step on this ruling, not pass a crash course.

The model can of course have its own manual, for audit and engineering. That manual does not answer the instance in front of the person concerned.

Why it happens

“Explainable AI” is often treated as a property of the model: linearisable, visualisable, drawable as a tree. That property is useful to researchers in the lab, because it answers how the apparatus generally works. The person concerned has a case-level question: on which facts of this case was a contestable conclusion reached. The two questions have different passing answers. One is a device manual; the other is a ruling.

Miller stresses that explanation is social communication about a particular event. The event here is this decision. Make the model the object and the topic of the exchange becomes engineering; the user cannot spend the words on submitting extra evidence, choosing to appeal, or accepting. “Already explained” is then true in the product log and false in the user’s task.

Studying it

Pair the same denial with a “model manual” (architecture, training data, overall accuracy) and a “decision account” (facts used in this case, conclusion, available actions). Watch what people choose next. Independent variables: object of the account (model / this decision), whether appeal is allowed. Dependent variables: whether they can name a changeable fact, relevance of appeal materials, rate of mistaking a model limitation for a case-level reason.

Overall accuracy will raise a feeling that “the system is solid” without improving the quality of the case-level action; report the two separately.

Where it stops holding

Procurement, oversight, and incident review need model-level accounts; this entry does not deny that audience. End users facing a high-stakes ruling need the decision level. Low-stakes creative suggestions barely have “a decision” to explain, and a ruling-style note will look absurd. This entry also does not treat how long the account should be — if the object is wrong, length will not save it.

Applying it

• For the person concerned, start with the facts of this case and end with an action they can take. Architecture, parameter counts, and training data go under “about this system,” not beside the result.
• If a model limit must be mentioned, write it as a limit on this conclusion (“image quality was insufficient; no malignant call was made this time”), not as product brochure.
• Offer two entries, “about this decision” and “how the system works,” and open the first by default.
• Check: give the copy to someone who has never seen the system and ask only “what should the applicant do next.” If they cannot answer, the object is still the model, not the decision.

Related

• Same group: L5.01.1 Explaining the process and explaining the outcome are different things · L5.01.2 A post-hoc explanation is not the true mechanism
• Nearby: L5.02 Local and Global Explanations · L5.05 The Moderation Principle of Transparency · L5.08 Counterfactual Explanations
• Search terms: decision-level explanation · model card versus decision · case-specific explanation