Validation should include actual operators, not only designer self-test

What it is

Representative operator participation requires that validation be carried out by real people who hold the target role, know the actual configuration, and span the relevant shifts and skill levels — not by having designers test their own work and calling that validation. Designer self-test is effective at catching implementation-level bugs, a button that does not respond, data that fails to refresh. But the designer already knows the design intent, the operating path, and what every label refers to, which disqualifies them from representing what a first-time user of the interface would actually struggle with.

Why it happens

A designer's mental model of their own interface is fully internalized — a form of the curse of knowledge: the designer can recognize instantly what an icon or menu item does, precisely because the design process itself was the process of forming that mental model. Every label and every layer of menu structure was a decision the designer made step by step, so for them there is no such thing as "not understanding it." That internalization cannot be reversed by the designer pretending to be a new user, because pretending does not erase what is already known — a designer can never genuinely experience where a person who has never seen the interface before would get stuck, or which label they would read as meaning something else. The value of testing with real operators goes beyond catching problems the designer never thought of: operators bring field-accumulated mental models and long-standing operating habits that often do not match the standard workflow the design specification assumed — the spec might assume operators check the overview before drilling into detail, while field habit is to judge from experience first and selectively verify afterward. This gap between the spec's assumptions and real working strategy can only be surfaced by real operators; reviewing the specification document, however carefully, will not reveal it.

Studying it

Plan recruitment across role, years of experience, shift, and relevant capability, and record the recruitment process and exclusion criteria honestly, rather than only inviting senior staff who are easy to reach and cooperative. Compare error paths across designers, representative operators, and edge-case users (new hires, people who use the function infrequently) — the three groups often get stuck at completely different points. If a participant receives excessive on-the-spot guidance or prompting during testing, that guidance itself should be recorded as part of the data rather than quietly erased and counted as the participant "completing it independently," because over-assistance contaminates any judgment about unaided performance; after a defect is fixed, retest with the same scenario to confirm the originally recorded difficulty was actually resolved rather than merely routed around.

Where it stops holding

Representativeness does not require the sample to be a statistical estimate of an entire industry's workforce — it requires covering the consequential variation that exists within this specific system: experience gaps between old and new configurations, day-shift versus night-shift differences, whether a particular specialized training was completed. Operator participation does not replace human-factors method or independent judgment — operators report "I got stuck here," but working out why, and what to change, still requires professional analysis. Participation itself needs to protect scheduling, informed consent, and freedom from being penalized on performance reviews for exposing operating difficulties during a test — otherwise future participants will be reluctant to reveal real difficulties honestly.

Applying it

• Recruit systematically from a role–task matrix, rather than inviting only the senior experts who are easiest to reach day to day.
• Have participants complete tasks with authentic tools, authentic procedure text, and authentic team coordination relationships, avoiding isolated testing detached from the real working environment.
• Record prompts, workarounds, and participants' spoken confusion as defect evidence, not just whether the task was eventually completed.
• Use a moderator who did not design the feature to run key validation tests, so tone and follow-up questions cannot inadvertently hint at the answer.

Related

• Same group: Y7.06.1 Verification and validation confirm that a new design meets safety goals in real operations · Y7.06.2 Normal, abnormal, and extreme scenarios must be covered · Y7.06.4 Validation results require traceable written records
• Nearby: Q2 Participants and sampling · Y6.03 Expert–novice differences
• Search terms: representative users · operator-in-the-loop · curse of knowledge