A comparison from observational data needs to name the variables it never controlled for
Aliases: confounding variables · limitations statement
What it is
Any between-group comparison from observational (non-experimental) data can be distorted by uncontrolled variables: comparing two channels' conversion rates, the channels' different user compositions (channel A naturally attracts high-intent users) are themselves an alternative explanation; comparing metrics across quarters, seasonality and macro conditions are uncontrolled variables. "This comparison does not control for X"—writing out the known-but-uncontrolled factors—is the key move that pulls an analysis back from "insinuating conclusions" to "honest evidence." Explicitly stating uncontrolled variables does not diminish a conclusion's value; it delimits the conclusion's scope, letting readers know under what conditions the comparison is trustworthy.
Why it happens
Stating uncontrolled variables works by making alternative explanations explicit: every observational comparison has logical competitors ("X caused the difference, not the cause you think"), and those competitors are either controlled (experimental design, statistical adjustment), acknowledged (a limitations statement), or hidden. Hidden, they are invisible to readers—who cannot gauge the conclusion's robustness and can only accept the literal wording. Explicit statement functions as triage: readers (or later analysts) can judge "is this confound fatal for my use?"—for some purposes (an initial directional read), mild known confounds are tolerable; for others (resource-allocation decisions), they must be controlled first. What needs stating is "known but uncontrolled" variables, which requires a frequently skipped step: actively inventorying which variables relate to both the independent and dependent variables (the definition of confounders) and judging each one's control status. The existence of that step is itself a quality watershed—an analysis that cannot name any potential confound usually means the inventory was never done, not that confounds don't exist.
Where it stops holding
Stating uncontrolled variables has an "over-disclosure" boundary: listing every conceivable confound unranked equals saying nothing (readers cannot tell which matter); effective limitations statements rank by impact and list the top few, with their direction ("channel A skews toward new users, possibly overstating true between-channel differences"). Statistical control (regression adjustment, matching) can handle some confounds, but adjustment itself rests on assumptions (no unmeasured confounding, correct model specification), so the statement must say "what was controlled and on what assumptions"—adjustment is not a declaration of no confounding. For experimental data (randomized controlled), randomization dissolves the uncontrolled-variable problem, and the statement's content shifts to "randomization unit and attrition bias"; the observational-confound template does not apply.
Applying it
- Attach a limitations statement to observational comparison charts, listing the major uncontrolled variables and their expected bias directions.
- Keep a confounder inventory in analysis documents: candidate variables relevant to the core comparison, each marked with its control status (controlled / uncontrolled / uncontrollable).
- For decision charts involving resource allocation, known uncontrolled confounds must be declared in the chart body, not a footnote.
- Verification: ask the analyst "if this conclusion is wrong, which uncontrolled variable is the most likely reason?" No answer means the confounder inventory was never done, and the chart's limitations statement is untrustworthy.