A p-value is compatibility of the data with the hypothesis, not the chance the effect is real

What it is

A p-value is the probability of a statistic at least as extreme as the one observed, computed under the null hypothesis and the statistical model in use. It measures compatibility of the data with that whole package—hypothesis plus model—not “the probability the effect is real,” not “the probability the null is true,” and not the colloquial “probability the result is due to chance.” Reading p = 0.03 as “97% chance this change is real” reverses the conditioning. Whether a significant gap is worth shipping is a separate decision, not this layer.

Why it happens

The tail probability is drawn from a model with the conditions already nailed down: the null holds, the sampling and independence claims hold, the measurements arise as declared. A small tail says only that, if this whole package were true, data like these would be uncommon. Uncommon can come from a treatment, or from a wrong model, an outlier, dependence treated as independence, or undeclared extra tests. p does not split those sources, and it does not emit a posterior table about the world. Turning p into “probability the effect is real” still needs a prior and a fully specified alternative; an ordinary significance test does not supply them. The same small p can therefore come from a true effect or a bad model, and the language usually names the former.

Studying it

Write p in the methods as a definition: tail probability under which null, which model, and which direction of extremity. Do not rewrite p as a posterior or as “confidence” in the abstract. Place model diagnostics (independence, clustering, distribution) next to p so readers can see what the compatibility is relative to. Resample under the null and check whether nominal p is calibrated; on failure, fix the model before telling a story that the effect got “more real.” If the question truly needs the probability a hypothesis is true, do an explicit Bayesian calculation with a public prior; do not dress p in posterior clothing.

Where it stops holding

On a narrow question with many replications and a familiar model, a small p is still a useful signal of incompatibility with the null model—and still not a posterior. Uncorrected p after an exploratory search is already a biased compatibility measure, and even less a measure of truth. Equivalence or minimum-effect tests ask a different hypothesis; the meaning of p travels with the hypothesis, and is not frozen as “is there an effect.” The teaching shorthand “probability it was chance” recurs in talk and has to be walked back to the definition each time.

Applying it

• Spoken template: “If there were no difference and the model were right, a result this extreme or more would show up about once in …”—never “there is a …% chance it is real.”
• In review, send back any sentence of the form “p is small, therefore the effect exists”; rewrite as compatibility and name the model assumptions.
• On a one-pager for decision-makers, put p in a footnote and magnitude plus interval in the body. If a probability sentence is required, run a Bayesian analysis; do not rename p.
• Check: circle every “probability it is true” and “high confidence” in the pack. Delete or rewrite any sentence that traces back to a p-value.

Related

• Same group: Q3.21.2 After the fact you cannot tell whether n was enough · Q3.21.3 A one-tailed test lowers the bar and must be declared first · Q3.21.4 Width of the interval tells precision better than whether it includes zero
• Adjacent: Q3.13 Significance and practical importance · Q3.22 Multiple comparisons and result picking
• Search terms: p-value · compatibility · posterior probability