New problem discovery declines with sample size but does not reach zero
Aliases: problem discovery curve · marginal discovery rate · saturation misconception
What it is
Diminishing returns in problem discovery means that, within similar users and tasks, common issues are found early and later participants increasingly repeat them. Marginal discovery usually declines but does not become zero: rare, segment-specific, and path-dependent failures can still appear later.
Why it happens
Cumulative discovery depends on each issue being triggered and recognized. High-probability issues appear early, leaving a pool dominated by low-probability ones. New tasks, product changes, or segments reset that pool. Correlated issues and participants also make simple independent-detection models optimistic.
Studying it
Plot new issues, cumulative serious issues, and repeats by participant, and use random reorderings to inspect order sensitivity. Keep merging rules, severity, and task coverage stable. Curves can compare rounds, but an unseen tail is not evidence of absence; combine critical-risk work with hazard analysis, telemetry, or targeted tests.
Where it stops holding
Saturation is conditional on sampling, tasks, and coding, not completeness of a product. Decline may also reflect moderator fatigue, weaker notes, or overaggressive issue merging. Old curves do not automatically predict a changed product, and pooled heterogeneous users can hide segment-specific failures.
Applying it
- Hold tasks and issue definitions stable within a round and track novelty, repetition, and severity.
- Before stopping, inspect uncovered segments, paths, and failure consequences.
- Restart the discovery curve after material product or task changes.
- Use risk-driven tests for high-consequence failures instead of waiting for chance discovery.