Small samples can expose many common severe usability problems
Aliases: small-sample discovery · severe usability problem · diminishing discovery
What it is
A few participants can sometimes reveal several common or easily triggered severe problems, supporting small-round formative testing. This does not guarantee discovery of most severe issues. Detection depends on users, tasks, state coverage, trigger probability, and recognition; rare high-consequence failures are readily missed.
Why it happens
High-probability issues recur early, creating diminishing novelty, while fix-and-retest cycles redirect samples to remaining risks. User paths and problems are correlated, making fixed independent-probability models optimistic. Severity follows consequence and context, not observation count.
Studying it
By round, record novel and repeated issues, severity, triggering task, and segment, and reorder participant sequences to inspect curve stability. Prespecify merging and severity rules. Add hazard analysis and targeted scenarios for safety, privacy, and critical business failure.
Where it stops holding
Small samples support discovery and iteration, not prevalence estimation or proof of exhaustion. Multiple segments, devices, and complex states expand coverage. No new issue may reflect weak tasks, fatigue, or overmerging rather than saturation.
Applying it
- Use small test–fix–retest rounds instead of one fixed-count delivery.
- Cover critical tasks, segments, and failure states and list omissions.
- Rate severity from consequences rather than frequency.
- Target rare severe risks separately and avoid unconditional “most problems found” claims.