Expert review cannot substitute for testing with target users
Aliases: expert prediction · user validation · method complementarity
What it is
Expert review predicts risk from knowledge; user testing observes target participants performing tasks. They provide complementary evidence. Inspection cannot replace evidence about goals, misunderstanding, workarounds, and affect, while small user samples cannot replace expertise on rare hazards and professional rules.
Why it happens
Experts use pattern libraries and normative reasoning; users contribute real knowledge, motivation, and constraints. Predictions can be false positives and observations cover only triggered states. Sequential use can remove obvious defects before users test remaining uncertainty, but provenance must survive early fixes.
Studying it
Map expert issues to user events and classify shared, expert-only, user-only, and untriggered findings. Preserve versions across fixes. For severe untriggered predictions, use targeted tasks, hazard analysis, or broader evidence rather than declaring absence.
Where it stops holding
Early low-fidelity work may temporarily use inspection alone, and urgent teams can fix obvious issues first; neither proves substitution. Expertise may suffice when the claim concerns standards compliance rather than use. User tests also fail with the wrong sample or tasks.
Applying it
- Assign each decision claim to expert, user, or combined evidence.
- Preserve provenance and never rewrite predictions as user findings.
- Validate recommendations affecting understanding and behavior with target tasks.
- Resolve conflicts by locating population, state, and consequence differences.