An overall impression induced from a few local explanations is often wrong

What it is

Three approved visa files in a row, each local account mentioning “complete itinerary.” The user then believes the system mainly looks at itinerary completeness. A fourth file with a complete itinerary is refused, and that is experienced as the system reversing itself. An overall impression induced from a few local explanations is often wrong: people are estimating the apparatus’s temperament from a small, biased sample of locals.

The error is not that those three local accounts were each false. It is the induction step.

Why it happens

People treat a word that keeps turning up as a rule. Local methods pick, near each instance, the factors that best round out this instance — often the ones that discriminate in that neighbourhood and are easy to say. Several instances in a row sharing one easy factor does not mean that factor dominates on the distribution. Factors not picked, and failures not shown, never enter the impression.

The interface further filters who sees an explanation: approved examples are opened more, forwarded more. The local sample that enters the induction then leans further toward a success story. LIME can be faithful to one instance and still silent on “what character this system has” — silence the user fills with induction.

Studying it

Show k locally explained examples (controllable: repeating factors vs diverse; successes only vs mixed), then have people estimate feature weights on the whole and predict an unseen file. Independent variables: k, factor repetition, success/failure mix, whether a true global slice is also given. Dependent variables: error on global weights, surprise on counterexamples, rate of judging unshown factors as “the system does not use that.”

If k grows but the source of the sample stays the same, error need not fall. Report where the sample came from, not only how many items.

Where it stops holding

If the model is close to globally linear and the local sample is random, induction can converge and this entry weakens. If users never make an “overall, how is it” judgement, the false impression is not spent and is temporarily harmless. Harm appears when they use it to guide the next file, or teach others a “how to pass” recipe. This entry does not void local accounts themselves.

Applying it

• After several local accounts in a row, do not let the interface hint “this is the rule.” If a sense of rule is needed, give a separately sampled global slice and say so.
• Mix in examples that contradict the current narrative when showing local accounts, to break “repeats therefore dominates.”
• Treat derived “tips from the explanations” with suspicion: they are usually local induction, not temperament.
• Check: have people who have used it several times write down “what it mainly looks at.” Compare that list with global weights computed on the distribution. Words that do not match are the false rules the local sample has been propping up.

Related

• Same group: L5.02.1 A local explanation accounts for a single output · L5.02.2 A global explanation accounts for overall behavioural tendency · L5.02.3 Users need different explanation levels in different situations · L5.02.4 Local explanations support accepting or contesting a single result · L5.02.5 Global explanations support judging whether to keep relying on the system · L5.02.7 Feature importance is a correlation ranking, not a causal account · L5.02.8 Developers need debugging information; end users need grounds for action — they cannot share one explanation
• Nearby: L5.01 Types of Explainability · L5.03 Trust Calibration · L5.09 Overtrust and Trust Collapse
• Search terms: local-to-global induction · biased sample of explanations · overgeneralisation from instances