Infer mental models from task observation; asking directly yields post-hoc stories

What it is

Asking “how do you think this system works” mostly yields a story cleaned up for a listener, not the structure that actually drove predictions during use. Inferred mental models from task observation put the evidence on how people act, where predictions fail, and how they repair, with talk used as a check. A post-hoc explanation can be fluent and reasonable and still contradict the operations just performed.

Why it happens

Causal stories during action are often local and incomplete, and people may not be able to retrieve them. An interview fills the fragments into a coherent narrative and rewrites them toward social expectation and terms just heard. Think-aloud during the task is closer to the prediction of the moment; “why did you do that” after the task starts justification. Inference from observation rests on publicly checkable traces: error types, ineffective repetition, ignored feedback, strategies that appear only in certain states. Those traces constrain the model so it cannot be merely a good speech.

Studying it

Primary chain: prediction points in the task recording (actions taken before feedback arrives), repairs after failure, states that were ignored. Place interview questions after the task, to check rather than to generate the model: when talk conflicts with behavior, keep the behavior and write the conflict as a finding. Do not explain “how it really works” before the task; that contaminates the later verbal model. Have a second analyst rebuild the model from de-identified behavior records only, then compare with the interview to measure which entities appear in talk but never in action.

Where it stops holding

Some knowledge can barely be anything but said—who can actually approve, which shortcuts must stay invisible to a manager. That is not a device model, but it still shapes strategy and needs other methods. Forcing commentary on hard-to-verbalize sensorimotor skill will manufacture a false model. Children and participants working across a language barrier are even poorer cases for interview-as-primary-evidence. Refusing all talk, on the other hand, throws away the words people use to name objects, and inference goes blind.

Applying it

• Mandate in the modeling process: draft entities and arrows from task behavior first, then open the interview as a check.
• On the findings page, color “relations supported by behavior” apart from “relations that appear only in talk”; the latter do not drive a redesign on their own.
• Replace “how does the system work” in the guide with “what do you expect to see next,” asked before the action.
• In review, play the failed-prediction clip and then show the diagram; if the diagram cannot explain the clip, change the diagram, not the spoken story.

Related

• Same group: Q4.14.1 A mental model diagram shows how users think the system works, not how it does · Q4.14.2 Gaps with the implementation model are where design must bridge · Q4.14.3 Experience levels can differ in kind, not only in amount
• Adjacent: Q2.07 Think-aloud · Q1.07 Researcher bias
• Search terms: verbal report · think-aloud · inferred mental model