Heterogeneous user populations require stratified sampling
Aliases: stratified sampling · subgroup coverage · oversampling
What it is
When capability, role, device, experience, or context relevant to an outcome varies substantially, analytic strata need separate evidence rather than one pooled headcount. Adding participants means supplying enough evidence for each group-specific question, not balancing every demographic label. More participants concentrated in the largest group do not fill a minority-group gap.
Why it happens
Pooling lets large groups dominate an average and can conceal opposing patterns. For independent stratum-specific estimates, precision is driven mainly by data in that stratum rather than the total headcount. Hierarchical models can use partial pooling under explicit exchangeability, shared-distribution, and model-form assumptions, stabilizing sparse estimates by sharing information across groups; they do not replace observations from an entirely missing group. In qualitative work, minority mechanisms and language can disappear under majority themes. Confirmatory subgroup comparisons should be defined before outcome inspection. Post-hoc grouping is legitimate exploration when labeled transparently and analyzed for multiplicity and sparse-cell instability.
Studying it
Choose strata from the decision and causal account, specifying whether the goal is within-stratum outcomes, between-stratum contrasts, or scenario coverage. Quantitative allocation follows desired precision, expected effect, and recruitment cost; analysis uses appropriate weights to recover population composition because oversampled raw proportions are not population estimates. Qualitative work can use quotas or maximum variation and assess information adequacy separately. Report under-recruited strata and unanswered questions.
Where it stops holding
Too many strata create sparse cells, unstable estimates, and high recruitment cost. Identity categories may be internally heterogeneous, sensitive, or dynamic and cannot replace mechanisms. A narrow homogeneous task may not benefit from stratification. Between-group comparison also requires measurement equivalence: the same scale or success rule may not mean the same thing across languages or assistive modes.