High-risk decisions require more than minimum-sample qualitative findings
Aliases: high-risk decision · evidence sufficiency · risk-driven research
What it is
Risk-based evidence planning increases evidential demands with the consequence, uncertainty, and irreversibility of a wrong decision. Small qualitative samples can reveal mechanisms and risk scenarios but usually cannot estimate prevalence, subgroup differences, or residual risk. A high-consequence decision is not justified merely because several interviews were completed.
Why it happens
Tiny samples are sensitive to common issues but can miss rare catastrophic events and small affected segments. Loss is asymmetric: missing one severe failure may cost far more than additional research. Evidence needs therefore follow consequences and decision thresholds, not the label of the method.
Studying it
Define false-adoption and false-rejection costs, minimum acceptable performance, and critical segments. Combine hazard analysis, qualitative mechanism discovery, controlled tasks, quantitative estimation, and deployment monitoring. Sensitivity analysis shows how unobserved risk rates change the decision. Stop rules should use risk coverage and uncertainty rather than headcount.
Where it stops holding
High risk does not require every study to be enormous. Strong mechanisms, prior evidence, conservative designs, and staged rollout can reduce data needs. Emergencies may require action under uncertainty; then prefer reversible choices and intensive monitoring. A large quantitative sample measuring the wrong construct provides no safety guarantee.
Applying it
- List failure consequences, affected groups, reversibility, and evidence gaps.
- Turn qualitative findings into testable risk scenarios and acceptance criteria.
- Use targeted tests, independent review, and staged release for severe outcomes.
- Predefine pause, rollback, and further-research triggers and verify them with operational data.