A counterfactual assumes other conditions held constant, which rarely holds in practice

What it is

“If income rose by four thousand this would pass,” on the model, usually means: change only the income cell, leave the others as they are. In a life, raising pay often moves debt ratio, housing cost, years in post, together. A counterfactual assumes other conditions held constant, which rarely holds in practice. The account is true in a sliced world and need not be true in a lived one.

Users revise as lives do, miss the slice, and experience the system as reversing itself.

Why it happens

The generator moves the fewest coordinates on an input vector. Coordinates are treated as independently assignable in training; in life they correlate. Income up pulls rent up, rent enters another cell, and the output does not walk the “income only” path. Wachter-style nearest change depends explicitly on that independent assignment; when the dependence is not said, users spend the sliced “if” as a life promise.

There is also time. The counterfactual is on the present model and the present other fields. By the time the user has revised and come back, the model may have changed and other fields may have moved. The promise is shorter-lived than the sentence looks.

Studying it

Place the change the counterfactual named into two simulations: “that cell only” and “correlated life changes together,” and see whether the flip is still there. Then have people revise a file as they understand it. Independent variables: strength of field correlation, whether “only if other items stay the same” is declared, whether the model is swapped between two applications. Dependent variables: agreement of flips across the two simulations, actual result after the user’s revision, rate of feeling deceived.

Low agreement while users still act on the single-cell promise is this entry happening in a product.

Where it stops holding

When fields are close to independent (a synthetic table in the lab, or a form where the user really can change one cell), the assumption approximately holds. Interactive probing (drag one cell, see the output at once, and see other cells rewritten in tandem) can make the assumption a visible constraint. Gamers who change only one cell on the form and leave the rest of life still, find the assumption holding for them — slice and life misaligned again, the other way.

Applying it

• Write the assumption: “the following holds only if other filled items stay the same, and only on the current version.” Do not write “if” as an unconditional promise.
• When joint change can be estimated, give a second line: “if income and rent move together at a common ratio, the result may still be a refusal.”
• Date the counterfactual and name the model version; withdraw or recompute after expiry or a version change.
• Check: find several real re-applications that followed the copy and also changed correlated items. If results disagree with the “one cell only” promise, put the assumption on the interface; if people still treat it as unconditional, the copy is not hard enough.

Related

• Same group: L5.08.1 A counterfactual explanation says which input changes would change the outcome · L5.08.2 The explanation is actionable only when the named input is one the user can change · L5.08.3 The same outcome has many counterfactuals; choosing which to show is itself a value judgement · L5.08.4 Counterfactual explanations leak decision rules and may be used to game them
• Nearby: L5.02 Local and Global Explanations · L5.01 Types of Explainability · L5.05 The Moderation Principle of Transparency
• Search terms: ceteris paribus · counterfactual assumptions · feature dependence