← Back to Statistics

Interpreting tests responsibly

p-values are widely misunderstood. Knowing what they do and do not mean protects you from misleading claims in news and research.

What a p-value is not

A p-value is not the probability that the null hypothesis is true, not the probability that the result happened by chance alone, and not a measure of the effect's size or importance. It is only the probability of data this extreme if \(H_0\) were true.

Significance thresholds are conventions

The 0.05 cutoff is a convention, not a law of nature. A p-value of 0.049 is not meaningfully different from 0.051. Many statisticians recommend reporting the exact p-value and an effect size with a confidence interval, and interpreting results in context rather than as a strict yes/no.

Effect sizes

An effect size describes how large a difference or relationship is, such as a mean difference in centimetres or a standardised difference (Cohen's d). Small effect sizes can be statistically significant with large samples, and large effects can miss significance with small samples.

The replication crisis

In psychology, medicine, and other fields, many published significant findings have failed to replicate. Causes include small samples, flexible analysis choices, multiple testing, and publication bias (journals preferring significant results). Pre-registering hypotheses and analyses before collecting data is one response.

Confidence intervals and tests agree

A 95% confidence interval contains exactly the null values that a two-sided test at \(\alpha = 0.05\) would not reject. So an interval for a mean difference of 2 to 8 units tells you the test rejects 'no difference' and also how big the difference plausibly is: more information than a p-value alone.

Worked example: reading a news claim

A headline says 'Coffee linked to longer life (p < 0.05)'. Ask: was it an experiment or observational study (confounding)? How large was the effect? How many outcomes were tested? Has it replicated? A single significant observational result is weak evidence of causation.

Bayesian alternatives

Bayesian methods, covered in the final unit, answer a different question: given the data, how probable are different parameter values? Some researchers prefer Bayes factors or posterior probabilities because they directly express belief in hypotheses, which p-values cannot do.

Common mistakes

Interpreting \(p = 0.03\) as a 3% chance that \(H_0\) is true. Treating 'not significant' as 'no effect'. Ignoring effect size. Trusting a single study without replication.

Key takeaways

A p-value measures surprise under \(H_0\), not truth or importance. Report effect sizes and confidence intervals. Beware of thresholds, multiple testing, and publication bias. Replication and pre-registration make findings more trustworthy.

Flashcard

p-value misreading

Not P(H0 is true)

It assumes H0 is true.

Check your understanding

A study reports p = 0.04. Which interpretation is correct?

← Previous lesson Next lesson →