Diminishing-return estimates assume independent evaluators with equal detection probability, conditions rarely met

What it is

Common evaluator-number formulas treat every problem as having a fixed detection probability and every evaluator as an independent sampler. In practice, probability varies with salience, task script, checklist item, and evaluator background; prior discussion, shared templates, or homogeneous teams violate independence.

Why it happens

Salient exceptions may be found by most people, while permission, concurrency, cross-page state, or domain violations require particular backgrounds. The same evaluator may be systematically strong in some areas and weak in others. Experience, training, and persuasion correlate results. Simple formulas can guide headcount but do not precisely predict coverage.

Studying it

State assumptions, interface scope, task scripts, and evaluator composition whenever reporting estimates. Estimate detection rates by problem category from empirical data rather than one global probability; analyze overlap matrices and stratified coverage. Sensitivity analysis shows how headcount conclusions depend on probability assumptions.

Where it stops holding

This does not make estimates useless. A coarse model still helps compare expected gains from two versus five evaluators. But do not extrapolate the curve into the total number of problems, or use one probability for salient errors and domain-specific errors.

Applying it

• Do not promise a single coverage figure; report found problems, overlap, and uncovered areas.
• Recruit evaluators stratified by severity relevance and domain category instead of assuming equal probability for all categories.
• Keep tasks, records, and initial severity ratings independent before aggregation.
• Supplement heuristic evaluation with logs, user testing, and production monitoring for the long tail.

Related

• Same group: B3.19.1 A single evaluator finds only a small part of all problems, and individuals differ widely · B3.19.2 Total findings show diminishing returns as evaluators increase, often flattening after three to five people · B3.19.4 Low overlap means the problem space is large and more evaluators are needed, not that quality is poor · B3.19.5 Evaluate independently before aggregating; prior discussion erases independence and distorts the number effect
• Nearby: Q4 Research Methods and Evaluation · Q2 Usability Evaluation
• Search terms: detection probability · coverage estimate · independence assumption