A model's internal consistency and its accuracy are two separate things — a consistent but wrong model still works

What it is

A mental model can be entirely "coherent" — every part fits together without contradiction, and using it to predict system behavior always produces a definite answer — while still being wrong overall, meaning it doesn't correspond to the system's actual mechanism at all. Internal consistency asks "does this story contradict itself"; accuracy asks "does this story match the facts." These are two independent dimensions — scoring high on one says nothing about the other.

Why it happens

The material users build a model from is a limited, fragmentary set of observations, not the system's design documentation. An explanation built on a completely wrong assumption can still cover everything the user has actually encountered coherently, as long as that assumption is applied consistently and never contradicted by a later observation — because "coherent" only requires no internal contradiction, it never checks whether the assumption itself corresponds to the real mechanism. Users have no second model to compare against, so a coherent model feels, from the inside, subjectively identical to a correct one — using either one feels exactly the same.

Studying it

Measuring the two dimensions separately is the key to telling them apart: consistency is tested by using the same model to make predictions across multiple situations and checking whether those predictions contradict each other; accuracy is tested by checking the model's predictions one by one against the system's actual behavior. Cross-tabulating the two results surfaces the "high consistency, low accuracy" case — the hardest state for a model to be caught in by introspection alone, and often the most stubborn kind, resistant to the negative feedback that ordinary use would otherwise provide.

Where it stops holding

This distinction only holds as long as the user has never run into a counterexample — the moment the system's real behavior exposes a mismatch with the model in some scenario, consistency takes an immediate hit, and the user either revises the whole model or patches it with an ad hoc exception to keep up the appearance of coherence. The latter path lets the model gradually degrade into a pile of stitched-together special cases, growing more complicated and less able to predict new situations with each patch — a degradation the user typically never notices happening.

Related

• Same group: A7.01.1 A mental model is a user's internal explanation of how a system works · A7.01.2 A model can be incorrect and still support successful operation · A7.01.3 The model determines a user's expectations and response when something goes wrong · A7.01.4 The core function of a mental model is predicting system behavior for a given action, not memorizing steps · A7.01.5 A sufficiently complete mental model lets a user derive operations they were never taught · A7.01.7 Users lack insight into the accuracy of their own model and rarely question it
• Nearby: A7.14 Identifying and correcting a wrong mental model · A7.12 Causal attribution and superstitious behavior
• Search terms: coherence versus correctness · internal consistency · folk theory