Users lack insight into the accuracy of their own model and rarely question it

What it is

Users almost never stop to ask themselves "is my understanding of this system actually correct" — as long as the model keeps producing usable predictions, they treat that understanding as settled fact rather than a hypothesis still open to question. This lack of insight isn't carelessness; it's a natural consequence of how mental models operate: the more smoothly a model runs, the less reason a user ever has to examine it.

Why it happens

Once a model is formed it recedes into the background of consciousness — what the user experiences is the prediction itself, not the process of applying a model, the same way a speaker doesn't consciously notice they're applying grammar rules while talking. Users have no external reference point: they don't know the system's actual mechanism, and there's usually no one else on hand to point out where they went wrong. Without counterexamples or an outside check, a model's internal consistency gets subjectively experienced as correctness, and the two are simply not distinguishable from the inside — so there's no trigger for doubt in the first place.

Studying it

A common way to quantify this blindness is to have users rate their confidence in some piece of understanding, then give them a task specifically designed to expose gaps in the model, and compare confidence against actual accuracy — confidence that's uncorrelated with, or even negatively correlated with, accuracy is direct evidence of this lack of insight. Another approach is simply asking "how sure are you that this is right" — most users report a confidence level well above the accuracy that later testing actually reveals.

Where it stops holding

This blindness isn't equally severe for every user or every situation — someone who has hit an obvious failure (an error traceable to their own mistaken understanding) markedly lowers their confidence in that specific piece of understanding, showing that concrete negative feedback can break through this blindness, though usually only for the small part of the model that the failure exposed, not the rest of it. Also, a complete novice may not even register that they've formed an understanding in the first place, which is a different kind of blindness from that of an experienced-but-never-challenged user — the former is closer to a blank slate, the latter is a false sense of certainty.

Related

• Same group: A7.01.1 A mental model is a user's internal explanation of how a system works · A7.01.2 A model can be incorrect and still support successful operation · A7.01.3 The model determines a user's expectations and response when something goes wrong · A7.01.4 The core function of a mental model is predicting system behavior for a given action, not memorizing steps · A7.01.5 A sufficiently complete mental model lets a user derive operations they were never taught · A7.01.6 A model's internal consistency and its accuracy are two separate things — a consistent but wrong model still works
• Nearby: A7.14 Identifying and correcting a wrong mental model · A7.06 Identifying a flawed model
• Search terms: illusion of explanatory depth · metacognitive blindness · overconfidence