Quantitative sample size depends on the effect worth detecting
Aliases: power analysis · smallest effect size of interest · precision planning
What it is
Effect-size-based sample size planning begins with the smallest decision-relevant difference and combines it with outcome variability, error or interval criteria, target power, and study design. Smaller effects usually require more observations to distinguish from noise. Effect size matters because fixed headcounts are unjustified, but no single effect number determines sample size by itself.
Why it happens
Precision generally improves with the square root of independent information, so halving error requires far more than doubling participants. Pairing, repeated measures, clustering, baseline covariates, event prevalence, and unequal allocation change effective information. Pilot effect estimates are often optimistic because of selection and random variation, producing underpowered follow-up. Planning around the smallest effect worth acting on is more defensible than using the most favorable historical point estimate.
Studying it
Specify the estimand, smallest important effect, plausible variance or baseline rate, allowable error, target power, and primary model, then use analytic calculation or simulation. Incorporate intracluster correlation, repeated measures, multiplicity, nonresponse, attrition, and invalid records. Estimation-focused studies can plan directly for interval width. Report every input and a sensitivity range rather than one assumption-free number.
Where it stops holding
Power analysis cannot repair biased sampling, invalid measurement, or misspecification. Post-hoc “observed power” based on the observed effect is usually a transformation of the p-value and does not explain a nonsignificant result. Huge samples can make trivial differences significant and increase privacy and cost. Rare events, exploratory models, and adaptive designs may require simulation, sequential rules, or precision targets rather than a simple two-group formula.
Related
- Same group: Q1.08.1 Discovery-rate heuristics have strict conditions · Q1.08.2 Heterogeneous populations require stratification
- Adjacent: Q3.21 Statistical significance and effect size · Q3.18 Experimental design and control
- Search terms:
power analysis·smallest effect size of interest·precision planning