A single evaluator finds only part of the available problem set
Aliases: evaluator effect · individual coverage · inspector variability
What it is
The evaluator effect is substantial variation in which problems reviewers detect and classify on the same interface. One reviewer generally covers only part of the discoverable set, but no fixed fraction applies. Expertise, path, time, heuristic interpretation, and issue definition shape coverage.
Why it happens
State space exceeds one inspection path and attention follows familiar domains. Usability experts may see principle violations but miss novice language; domain experts understand consequences but normalize local practice. More homogeneous reviewers repeat blind spots, so complementary backgrounds matter.
Studying it
Build evaluator-by-issue matrices for unique findings, overlap, severity disagreement, and incremental discovery. Randomize starts or allocate coverage to avoid confounding with navigation order. When comparing user data, separate untriggered issues from genuine false positives.
Where it stops holding
One reviewer can provide useful early screening. Higher issue count does not prove skill because one cause may be fragmented. Multiple reviewers still do not establish completeness for rare states or longitudinal use.
Applying it
- Select complementary domain, accessibility, and novice perspectives.
- Allocate tasks and states rather than duplicating one path.
- Preserve unique severe findings during consolidation.
- Route residual blind spots to user tests, telemetry, or targeted risk review.