Exploratory data cannot independently confirm the hypothesis they generated
Aliases: circular analysis · double use of data · data reuse bias
What it is
Discovering a pattern in a dataset and then treating the same pattern in the same data as independent confirmation is double dipping or circular analysis. Exploration is not the error. The error is concealing that the hypothesis was data-derived and presenting “these data suggested an account” as “new evidence supported the account.” Writing a post-hoc account as if it were an a priori prediction is commonly called HARKing.
Why it happens
Exploration selects the most striking result from many measures, subgroups, time windows, or explanations. The selected pattern contains both signal and sample-specific chance variation. Reusing that same variation to estimate evidential strength creates selection bias, inflates effects, and invalidates ordinary error rates. It is equivalent to choosing the densest cluster of shots, drawing a target around it, and using those shots to demonstrate accuracy.
Studying it
For confirmatory or predictive claims, the direct remedy is independent replication. When data are expensive, split exploratory and confirmatory sets before analysis, or nest model selection inside cross-validation when estimating generalization; do not repeatedly inspect the holdout before freezing the rules. Without new evidence, label the analysis exploratory and disclose the search space, candidate analyses, and sensitivity checks rather than claiming independent confirmation. Qualitative traditions have different aims and evidential logics: negative-case comparison, theoretical sampling, or new cases can challenge and refine an interpretation when appropriate, but they are not a universal requirement to create a separate confirmation set.
Where it stops holding
Data need not literally be used only once. Prespecified summaries, model fitting, and internal diagnostics can share a dataset. The problem is presenting a selected pattern as unselected independent evidence. Splitting reduces information and can be inefficient with small samples, making transparent exploration followed by later replication preferable. In cross-validation, data-driven feature selection and parameters for scaling, imputation, or learned representations must be fitted inside each training fold and then applied to its test fold. A prespecified per-sample deterministic transformation that estimates nothing from test data does not become leakage merely because it is applied to every sample. HARKing is related but not synonymous: it hides when a hypothesis was formed, whereas double dipping lets evaluated data participate in selection or model construction.