Statistics › Section J · Hypothesis testing with two samples › Topic J.2

Effect sizes and reading tests responsibly

"p < 0.05" is where the public story starts and the statistician's questions begin. Is the effect large enough to matter? Was the design honest? Would it replicate? This topic builds the habits that separate reading a significant result from trusting one.

Beyond the threshold

"p < 0.05" is where the public story starts and the statistician's questions begin. A p-value measures surprise under the null - not truth, not importance, not size. This topic builds the tools that fill the gap: effect sizes that say how large, intervals that show the plausible range, and the reading habits - thresholds, multiple tests, publication bias, replication - that keep significant-sounding claims honest.

Effect sizes

An effect size describes how large a difference or relationship is. The most honest version is raw units: a tutoring method worth 6 points, a drug that lowers blood pressure by 7 mmHg - numbers a reader can weigh directly. Standardised versions exist to compare across studies and scales, expressing a difference in standard-deviation units. The two questions every report should answer are therefore: is there convincing evidence of an effect, and how big is it? A p-value answers only the first.

Cohen's d

The standardised effect size for a difference in means is Cohen's \(d\): the difference divided by a standard deviation,

d = \frac{\bar{x}_1 - \bar{x}_2}{s_{\text{pooled}}}

It counts how many SDs apart the groups' centres sit, making studies on different scales comparable. Commonly used benchmarks: around 0.2 is called a small effect, 0.5 medium, 0.8 large - conventions for orientation, not laws of nature, and fields differ: 0.5 SDs of reading improvement would transform a classroom, while 0.5 SDs of headache-relief days might not justify a drug's side effects.

Worked example: the tutoring effect, sized

The two-method comparison from the previous topic: difference 6 points, \(s_1 = 8\), \(s_2 = 9\), \(n_1 = 25\), \(n_2 = 30\). 1. Pooled SD:

s_{\text{pooled}} = \sqrt{\frac{24 \times 64 + 29 \times 81}{53}} = \sqrt{\frac{3885}{53}} \approx 8.6

2. \(d = \frac{6}{8.6} \approx 0.70\): a medium-to-large effect by the conventional benchmarks. 3. The full story: statistically significant \(p \approx 0.012\) AND \(d \approx 0.70\), with interval (1.4, 10.6) - evidence that a real, educationally meaningful difference favours method A. Check: the interval's lower bound corresponds to \(d \approx 0.16\), its upper to \(d \approx 1.24\) - size and precision in one statement.

Significance is not importance

Small effects reach significance with large samples; large effects miss it with small ones. A diet that lowers weight by 0.1 kg might produce \(p < 0.001\) in a study of 100,000 people - a triumph of sample size over substance that would matter to no one. Conversely, a genuinely large effect studied with 12 subjects can land \(p = 0.08\) and be dismissed. Significance answers "is it there?"; the effect size and interval answer "does it matter?" - and only the second question justifies action.

Thresholds are conventions

The 0.05 cutoff is a convention, not a law of nature: \(p = 0.049\) and \(p = 0.051\) are nearly identical evidence on opposite sides of a line. Rounding the p-value into "significant / not" throws away exactly the information that would prevent a false cliff-edge. The better practice: report the exact p-value, the effect size, and a confidence interval, and let readers judge the strength in context. "p = 0.12, difference 8 points" is often more informative than "significant at 0.05, difference 0.3 points".

Intervals and tests agree

The CI-test agreement from the intervals chapter now does daily work: a 95% interval contains exactly the null values that a two-sided 0.05-level test would not reject. An interval for a mean difference of 2 to 8 units therefore reports the test's verdict (0 excluded - the null of no difference is rejected) AND the size (the difference plausibly runs 2 to 8 units) in one statement. This is why applied reporting increasingly leads with intervals: they subsume the test and answer the question the test cannot - how big.

The replication crisis

In psychology, medicine and other fields, many published significant findings have failed to replicate - re-run studies found smaller effects, or none. The causes are structural: small under-powered samples, flexible analysis choices (deciding what to test after seeing the data), multiple testing - 20 tests at \(\alpha = 0.05\) with no real effects still expect about one "significant" result - and publication bias, since journals prefer significant findings, so the literature over-represents lucky draws. Responses include pre-registration (declaring hypotheses and analyses before the data arrive), corrections like Bonferroni (dividing \(\alpha\) by the number of tests), and valuing replications as publications in their own right.

Worked example: reading a news claim

A headline: "Coffee linked to longer life (p < 0.05)". Interrogate it in four questions. 1. Design: experiment or observational study? Coffee drinkers differ from abstainers in exercise, income, sleep - confounding, not causation, is the default explanation for an association. 2. Size: how much longer? A few weeks of life expectancy is "linked to longer life" at the headline's precision; five years is a different story. 3. Multiplicity: how many outcomes and subgroups were tested? A significant result among 30 comparisons is fishing, not finding. 4. Replication: has anyone re-run it? A single significant observational result is weak evidence of causation.

A Bayesian footnote

Every test in this course measures surprise under a hypothesis; a p-value never says how probable the hypothesis is given the data. Bayesian methods answer that second question directly: given the data, how probable are different parameter values? Some researchers prefer Bayes factors or posterior probabilities because they express degrees of belief, which p-values do not. That framework gets its own treatment in this course's final section - the point here is that the responsible reader knows which question a number answers before trusting it.

Common mistakes to avoid

First, reading \(p = 0.03\) as a 3% chance that \(H_0\) is true - the definitional error. Second, treating "not significant" as "no effect": under-powered studies miss real effects, and the Type II topic's warning applies. Third, ignoring effect size: no p-value, however small, says the effect matters. Fourth, treating 0.05 as a wall rather than a convention. Fifth, trusting a single study - significant results from small samples are the most likely to be overstated. Sixth, forgetting confounding when the design is observational: significance is evidence of an association, not of why it exists.

Key takeaways

A p-value measures surprise under \(H_0\) - never the probability \(H_0\) is true, never the effect's size or importance. Report effect sizes (raw units, or Cohen's \(d\) with the 0.2/0.5/0.8 conventions as orientation) and confidence intervals, which subsume the test and size the effect: the tutoring study's \(d \approx 0.70\), interval (1.4, 10.6). The 0.05 threshold is a convention; exact p-values beat cliff-edges. The replication crisis has structural causes - small samples, flexible analyses, multiple testing (Bonferroni's \(\alpha/k\) as one fix), publication bias - and structural remedies: pre-registration and replication. Read headlines with four questions: design, size, multiplicity, replication. Bayesian methods, in the final section, answer the probability question p-values cannot.

Flashcards

Flashcard

Effect size

How large a difference or relationship is

The question p-values never answer.

Test your understanding

Question 1 of 5

A study reports p = 0.04. Which interpretation is correct?

Question 2 of 5

A tutoring difference is 6 points with pooled SD about 8.6. What is Cohen's d?

Question 3 of 5

By the conventional benchmarks, d = 0.5 is called:

Question 4 of 5

A study of 100,000 people finds a 0.1 kg diet effect, p < 0.001. The best reading:

Question 5 of 5

An observational study links coffee to longer life (p < 0.05). What is the default explanation for the association?

← Previous topicNext topic →