Feature importance is a correlation ranking, not a causal account

What it is

The ranking’s first row says “age.” The user reads “you were refused because you are old.” What the ranking did is: in this score, or in this batch, the age variable moved with the output more than others. Feature importance is a correlation ranking, not a causal account. Moving together is not “change this and the result changes,” and still less “the system is treating you on this ground.”

Spend the ranking as a reason, and both appeals and revised materials will aim at the wrong object.

Why it happens

Importance scores come from perturbation, decomposition, or weights in a surrogate: how hard the output swings when a column is dropped or shuffled. A hard swing means that column carries information. The information can be a real cause or a stand-in — age standing for grade, postcode for income, word frequency for register. The ranking does not promise an intervention. The user’s default reading is intervention: “important” equals “if I change it I can reverse this,” and also equals “they are discriminating on this.”

Both local and global methods can emit this ranking. Scope does not settle the issue. LIME’s local weights are likewise neighbourhood correlation, not a guarantee that “if you replace this pixel the label must flip” — that is a different question.

Studying it

Show the same importance list, with or without a “this is association, not cause” note, then ask two things: if only this item is changed, will the output move; is the system treating the applicant on this ground. Independent variables: visual of the list (bar ranking / causal arrows), the note, whether an interventional contrast is also shown. Dependent variables: rate of causal misreading, actual success after people change inputs along the ranking, sense of unfairness.

The real intervention has to run the model. Do not substitute “I think it would move” from a questionnaire.

Where it stops holding

When factors are independent and importance comes from a true intervention experiment, the ranking can approach cause and this warning narrows. If users treat the list only as “clues worth checking,” not “change it and reverse,” the cost of misreading drops. If the ranking is drawn as arrows from features into the decision, a causal reading is almost guaranteed. This entry does not discuss whether to give local or global scope.

Applying it

• Title it “items that moved more with this output,” not “reasons” or “because.” Do not draw the bars as arrows into the result.
• If an item cannot be intervened on (age, a proxy for ethnicity), write plainly that it cannot be changed and should not be treated as something the person is supposed to go and change.
• When you need an actionable causal sentence, run an intervention or a counterfactual; do not retitle the importance list and pass it off.
• Check: point at the top row and ask “if only this item changed, what would happen to this result” and “is the system treating you this way because of this item.” Two yeses, and you have no intervention evidence, means the ranking is being spent as a causal manual.

Related

• Same group: L5.02.1 A local explanation accounts for a single output · L5.02.2 A global explanation accounts for overall behavioural tendency · L5.02.3 Users need different explanation levels in different situations · L5.02.4 Local explanations support accepting or contesting a single result · L5.02.5 Global explanations support judging whether to keep relying on the system · L5.02.6 An overall impression induced from a few local explanations is often wrong · L5.02.8 Developers need debugging information; end users need grounds for action — they cannot share one explanation
• Nearby: L5.08 Counterfactual Explanations · L5.01 Types of Explainability · L5.05 The Moderation Principle of Transparency
• Search terms: feature importance · correlation is not causation · attribution versus intervention