Low overlap means the problem space is large and more evaluators are needed, not that quality is poor

What it is

Low overlap is not inherently a failure. It may mean interface problems are spread across modules, roles, and input conditions, or that evaluator backgrounds are complementary and each sees a different layer. Interpreting overlap requires distinguishing complementary findings, false positives, and task-coverage differences.

Why it happens

High overlap is common for salient, frequent, or surface problems; low overlap can come from long-tail issues, domain rules, asynchronous state, and cross-page paths. If each person’s findings are real and unique, the current headcount has not covered the problem space. Pressing everyone to “find the same problems” encourages chasing visible defects and missing important hidden failures.

Where it stops holding

Low overlap can also mean evaluators ignored the script, interpreted the checklist differently, or reported preference rather than defect. Verify facts and severity before interpreting overlap. Some safety-critical reviews require high overlap to confirm the same defect rather than accepting low overlap as evidence of complexity.

Applying it

• In the aggregation meeting, verify evidence for each unique finding before labeling it complementary, duplicate, or false.
• Plot an evaluator-problem matrix to locate modules and categories covered only by specific backgrounds.
• When low-overlap findings are real, add evaluators from the corresponding domain, platform, or accessibility area.
• Report why overlap is low: large problem space, differing scripts, ambiguous checklist, or insufficient training.

Related

• Same group: B3.19.1 A single evaluator finds only a small part of all problems, and individuals differ widely · B3.19.2 Total findings show diminishing returns as evaluators increase, often flattening after three to five people · B3.19.3 Diminishing-return estimates assume independent evaluators with equal detection probability, conditions rarely met · B3.19.5 Evaluate independently before aggregating; prior discussion erases independence and distorts the number effect
• Nearby: Q2 Usability Evaluation · Q4 Research Methods and Evaluation
• Search terms: overlap analysis · problem space · coverage gap