Statistics › Section I · Hypothesis testing with one sample › Topic I.4

Rare events, p-values, and the decision

The courtroom has heard the claim and measured the evidence: the data sit z standard errors from what the null predicts. The p-value converts that distance into surprise - how often would a result this extreme happen if the null were true? - and rare results indict assumptions.

The surprise meter

The courtroom has heard the claim, and the evidence is in: the test statistic says the data sit a measured distance from what the null predicts. The p-value converts that distance into surprise: how often would a result this extreme happen if the null were true? A rare result under \(H_0\) is the rare-event logic of the CLT chapter, now formalised: when the assumed world fails to produce the data, the assumption stands indicted.

The p-value, defined

The p-value is the probability, computed assuming \(H_0\) is true, of obtaining a result at least as extreme as the one observed. Every word carries weight: assuming \(H_0\) - the calculation lives in the null's world; at least as extreme - not exactly this result, but this extreme or more; and observed - the data set the bar. A small p-value says results like this would surprise a true null, which is evidence against it. A large p-value says only that nothing surprising happened: an absence of evidence against the null, not evidence for it.

What the p-value is not

The p-value is not the probability that \(H_0\) is true - \(H_0\) is either true or not; it gets presumed, not assigned probabilities. It is not "the probability the result was due to chance", a phrase that reverses the logic: the p-value assumes chance fully in charge and asks how often such data would appear. It is not the probability of a Type I error - that is \(\alpha\), fixed in advance. Reading p as "the probability the null is right" is the most common and most damaging error in reading research.

Computing p: three forms

The alternative's form sets the tail. \(H_a: \mu < \mu_0\): left tail, \(p = P(Z < z)\). \(H_a: \mu > \mu_0\): right tail, \(p = P(Z > z)\). \(H_a: \mu \ne \mu_0\): both tails, \(p = 2 \times P(Z > |z|)\) on the symmetric z curve. The one-sided/two-sided choice was fixed in the first topic of this section, and it matters exactly here: the same data give a two-sided p-value twice the one-sided one, which can straddle the \(\alpha\) line. Form first, p-value second, always in that order.

Worked example: the cereal test, concluded

The cereal test stands at \(z = -2\), with \(H_a: \mu < 500\). 1. One-sided, left tail: \(p = P(Z < -2) = 0.0228\). 2. Compare: \(p = 0.0228 \le \alpha = 0.05\), so reject \(H_0\). 3. Conclusion, in context: the data provide convincing evidence that this brand's boxes average less than 500 g - the consumer group's suspicion is supported. 4. Check: fair filling would produce a result this extreme in only about 2.3% of samples - rare enough to reject - and the conclusion names the direction, the units and the strength of evidence.

Worked example: the senator's poll, concluded

The poll's statistic is \(z = 2\); suppose the alternative is two-sided, \(H_a: p \ne 0.5\). 1. \(p = 2 \times P(Z > 2) \approx 0.0455\). 2. Compare: \(0.0455 \le 0.05\) - reject \(H_0\), barely: evidence of a difference from 50%, statistically significant, with the data pointing upward. 3. Check: the same data tested one-sided would give \(p = 0.0228\) and reject comfortably - which is exactly why the form is fixed before the data arrive. A verdict must not depend on choosing the yardstick after the measurement.

The decision and its grammar

The rule is one line: if \(p \le \alpha\), reject \(H_0\) - the result is statistically significant at level \(\alpha\). If \(p > \alpha\), fail to reject \(H_0\): not enough evidence. The grammar matters as much as the arithmetic. Never "accept \(H_0\)"; never "prove \(H_a\)": a test weighs evidence, it does not certify truth - "not guilty" is not "innocent", and a non-rejection is not proof that the null holds. A proper conclusion reads "the data provide convincing evidence that...", names the direction, and returns to the context the question came from.

Choosing alpha, once, before the data

The significance level \(\alpha\) is the false-alarm rate the study agrees to tolerate - the Type I error topic's price tag - and it is chosen before the data arrive: 0.05 in most scientific practice, 0.01 where false alarms are costly, 0.10 in exploratory work. Choosing \(\alpha\) after seeing the p-value is moving the goalposts: the standard stops being a standard and becomes a rubber stamp. When a report says "significant at the 0.05 level", it promises the bar was set in advance and the evidence cleared it - a promise whose whole value rests on being made first.

Common mistakes to avoid

First, reading p as the probability that \(H_0\) is true - the definitional error. Second, "accepting" \(H_0\) on a large p-value: absence of evidence is not evidence of absence. Third, switching the alternative's form after the fact to move a result across 0.05. Fourth, choosing \(\alpha\) after seeing p. Fifth, reporting "significant, p = 0.049" with no effect size - significance is not importance, a warning the effect-sizes topic develops. Sixth, writing "the boxes were proven underfilled": tests weigh evidence; they do not prove.

Key takeaways

The p-value is the probability, assuming \(H_0\), of a result at least as extreme as observed - a small p means the data surprise the null, which is evidence against it, never the probability that \(H_0\) is true. The alternative's form sets the tail: left, right, or doubled for two-sided - the cereal's \(p = 0.0228\), the poll's \(p \approx 0.0455\). Decide by the rule \(p \le \alpha\) rejects, with \(\alpha\) fixed before the data; report "statistically significant" only when that promise holds. Conclusions weigh evidence in context - convincing, directional, unproven - and never accept the null.

Flashcards

Flashcard

p-value

P(result at least as extreme | H0 true)

The surprise meter.

Test your understanding

Question 1 of 5

The p-value of a test is:

Question 2 of 5

A test gives p = 0.003. At α = 0.05, the correct conclusion is:

Question 3 of 5

With z = 2 and a two-sided alternative, the p-value is:

Question 4 of 5

A test gives p = 0.07 at α = 0.05. The decision is:

Question 5 of 5

A study reports p = 0.04. The correct reading:

← Previous topicNext topic →