Counterfactual explanations leak decision rules and may be used to game them

What it is

The interface says “one fewer credit card and this passes.” The next batch of applications starts parking cards in a side account. The score passes; the risk does not change. Counterfactuals leak decision rules and can be used to game them. The more actionable the account, the more it is a walkthrough.

Gaming is not simply the user being dishonest. The system drew its boundary on changeable fields; changing the fields is the rational response.

Why it happens

A counterfactual exposes the local normal to the boundary: which way to walk to flip. Enough local normals can be pieced back into a rule, or into an operating list that is good enough. Applicants, intermediaries, and crawlers will collect those lists. The list is optimised for passing, not for the situation improving — income did not rise, the form did.

The virtue of a Wachter-style account (actionable, concrete) is the attack surface. A vague global temperament (“the system is strict”) is hard to spend on a form; “one fewer card” is easy. Higher actionability, cheaper gaming. This is not “which one to present carries value”: there the designer is choosing a life for the user; here the user is spending the chosen road as a hole.

Studying it

On a task that can be applied for again, supply counterfactuals and watch how later applications’ fields move toward the direction the copy named, and whether true risk labels improve in step. Independent variables: how concrete the counterfactual is, whether rate-limited, whether only a direction is given and not a threshold. Dependent variables: rate of field manipulation, gap between pass-rate rise and risk improvement, rate of the copy’s original sentence appearing in intermediary scripts.

A one-shot lab task will understate gaming, because it has no return from “apply again.”

Where it stops holding

One-shot decisions that cannot be resubmitted (this medical-record label) have a narrow gaming window. When the adversary is already strong (professional intermediaries), they will probe without counterfactuals; giving them only lowers the cost of probing. Giving no actionable account at all shuts off legitimate repair with the gaming; a window has to be taken between repair and gaming, rather than pretending the tension is not there. Whether other conditions held constant can make gaming fail in the real world — that is the next cut.

Applying it

• Fields at high manipulation risk (debt that is easy to hide, channels that can be dressed up) do not get a concrete threshold. Give a direction, or a proof that must be verified, not “one fewer.”
• Rate-limit the same applicant: a counterfactual is not a console for iterating on the boundary.
• Watch for abnormal shifts in field distributions after the account appears. A shift with no movement in risk labels means gaming is already happening.
• Check: an internal red team revises applications as the copy says, without improving the real situation. If pass rate rises clearly, this account is a walkthrough; remove the concrete threshold or change it to a repair that must be verified.

Related

• Same group: L5.08.1 A counterfactual explanation says which input changes would change the outcome · L5.08.2 The explanation is actionable only when the named input is one the user can change · L5.08.3 The same outcome has many counterfactuals; choosing which to show is itself a value judgement · L5.08.5 A counterfactual assumes other conditions held constant, which rarely holds in practice
• Nearby: L5.05 The Moderation Principle of Transparency · L5.02 Local and Global Explanations · L3.02 Source Attribution
• Search terms: counterfactual gaming · recourse and gaming · decision boundary leakage