Total findings show diminishing returns as evaluators increase, often flattening after three to five people

What it is

Adding evaluators usually adds unique problems, but marginal returns diminish: the first through third evaluators often add the most, and the curve commonly flattens after three to five. The number of evaluators should match risk, stage, and available resources rather than seek everyone.

Why it happens

Common problems are easier for multiple people to overlap on, so early evaluators quickly cover salient paths; later evaluators increasingly find long-tail, infrequent, or experience-dependent issues while aggregation and deduplication costs rise. Complex enterprise interfaces, cross-platform products, and safety-critical contexts have larger problem spaces, so the curve flattens later.

Studying it

Record unique problems added as each evaluator joins and plot the cumulative discovery curve; stratify by severity to see when high-severity findings saturate. Compare increments from domain, platform, and accessibility backgrounds. Use the marginal curve to decide whether to add evaluators or move to user testing and log monitoring.

Where it stops holding

“Three to five” is not universal; it depends on complexity, task breadth, checklist quality, and evaluator independence. If findings remain highly nonoverlapping, five may be too few; for early rough inspection, two or three may suffice. Quality depends on backgrounds and process, not count alone.

Applying it

• Use two or three independent walkthroughs early; use three or more with complementary backgrounds before release or on hazardous paths.
• Plot new findings after aggregation; stop adding evaluators when two consecutive people add little, then move to real-task testing.
• Add focused reviews for uncovered modules, roles, and platforms.
• Record each person’s unique findings and overlap as evidence for the next round’s number and composition.

Related

• Same group: B3.19.1 A single evaluator finds only a small part of all problems, and individuals differ widely · B3.19.3 Diminishing-return estimates assume independent evaluators with equal detection probability, conditions rarely met · B3.19.4 Low overlap means the problem space is large and more evaluators are needed, not that quality is poor · B3.19.5 Evaluate independently before aggregating; prior discussion erases independence and distorts the number effect
• Nearby: Q2 Usability Evaluation · Q4 Research Methods and Evaluation
• Search terms: diminishing returns · evaluator number · coverage curve