Statistics › Section I · Hypothesis testing with one sample › Topic I.2
Type I and Type II errors
No test can guarantee a correct verdict. Knowing the two ways a test can go wrong, and what each costs, is how studies choose their significance level and their sample size.
Two ways to be wrong
A hypothesis test ends in a verdict: reject the null, or fail to reject. But the null's truth is unknown - it may be true or false regardless of the verdict - so four outcomes are possible, and two of them are errors. This topic names the two, prices them, and faces the uncomfortable seesaw between them: push one error down and the other rises, unless the sample grows.
The four outcomes
Line up the null's actual state against the verdict. \(H_0\) true, fail to reject: correct. \(H_0\) false, reject: correct - a detection. \(H_0\) true, reject: error - a false alarm. \(H_0\) false, fail to reject: error - a missed detection. The four cells are the whole picture; every test ends in one of them, and no analysis can announce which, because the truth is the very thing being tested. Statistics can only set the error rates honestly.
Type I: the false alarm
Rejecting a true \(H_0\) is a Type I error: the data looked convincing, but nothing was really going on. Its probability is \(\alpha\), the significance level - set in advance, 0.05 in much of practice, 0.01 where false alarms are costlier. In the courtroom, Type I is convicting the innocent - the error legal systems are built to avoid, which is why "beyond reasonable doubt" sets a small \(\alpha\). In a drug trial, Type I approves a medicine that does not work.
Type II: the missed detection
Failing to reject a false \(H_0\) is a Type II error: something real was going on, and the data failed to show it. Its probability is \(\beta\), which - unlike \(\alpha\) - depends on things the analyst does not fully control: the sample size, the size of the true effect, the noise in the data. In the courtroom, Type II frees the guilty; in a drug trial, it shelves an effective treatment. Misses do not announce themselves: the study ends "not significant", and a real effect walks away unrecognised.
The seesaw
At a fixed sample size, \(\alpha\) and \(\beta\) move oppositely. Tighten the standard for rejection - demand stronger evidence, lower \(\alpha\) - and more real effects slip through: \(\beta\) rises. Loosen it, and false alarms multiply. The seesaw is mathematical, not a failure of effort: one knob turns both rates in opposite directions. The escapes are to buy more data, since larger \(n\) lowers \(\beta\) at any \(\alpha\), or to pick the point on the seesaw where the costs balance for the decision at hand.
Worked example: which error should each device fear?
The costs decide, and they are contextual. Take each device's null as "nothing is wrong": the message is genuine, there is no fire. A spam filter's Type I - real mail sent to the junk folder - can lose a job offer; its Type II merely shows an advertisement: tune toward tolerating Type II. A smoke alarm's Type I wakes a household unnecessarily - annoying; its Type II fails to wake a burning house - catastrophic: accept extra false alarms to make misses rare. Courtrooms price a wrongful conviction far above a wrongful acquittal; a safety inspection may reverse the ranking. Statistics sets the rates; the decision context sets their worth.
Power: the detection rate
Power is the probability the test does what it exists to do: reject a false \(H_0\), correctly detecting a real effect. Power is \(1 - \beta\), the mirror of the miss rate. It rises with four levers: larger samples (less blur around the truth), larger true effects (louder signals), less variability (quieter backgrounds), and a larger \(\alpha\) - the seesaw again. Studies are commonly planned for 80% power: an honest acknowledgement that detection is imperfect, made in advance, so a null result can be read as "no effect detectable at this size" rather than "no effect".
Worked example: the four cells of a drug trial
A trial tests \(H_0\): the new drug adds nothing. a) The drug truly adds nothing, and the trial finds no effect: fail to reject - correct. b) The drug truly works, and the trial finds significance: reject - correct, a detection. c) The drug truly adds nothing, but a fluke sample looks impressive: reject - Type I, and a useless drug reaches patients. d) The drug truly works, but the sample is small and the signal drowns: fail to reject - Type II, and a good treatment is shelved. Check: the two correct outcomes depend on the drug's real effect, which is unknown; the two errors are priced by \(\alpha\), chosen in advance, and by \(\beta\), designed against through power planning.
Common mistakes to avoid
First, treating the two errors as symmetric - their costs are set by the context, not the mathematics. Second, trying to lower both at a fixed sample size: the seesaw forbids it. Third, forgetting that \(\beta\) and power depend on the true effect's size, which is unknown - hence planning targets like 80% for effects considered worth detecting. Fourth, reading "not significant" as proof of no effect: that reading is Type II's error made into philosophy. Fifth, confusing power, \(1 - \beta\), with the significance level \(\alpha\). Sixth, raising \(\alpha\) after the fact to force significance - the seesaw tipped after the verdict.
Key takeaways
Four outcomes, two errors. Rejecting a true \(H_0\) is Type I, with probability \(\alpha\), the significance level set in advance. Failing to reject a false \(H_0\) is Type II, with probability \(\beta\). At fixed \(n\) they seesaw - lower one, raise the other - and more data or less noise is what lowers both together. Which error to fear is a cost question: spam filters fear Type I, smoke alarms fear Type II, courtrooms fear Type I most. Power, \(1 - \beta\), measures detection and is planned before the study - commonly toward 80% - so that "not significant" can be read honestly.
Flashcards
Flashcard
H0's truth × the verdict
Test your understanding
Question 1 of 5
A smoke alarm's null is "there is no fire". The alarm sounds, but nothing is burning. Which outcome is this?
Question 2 of 5
A Type II error is:
Question 3 of 5
The significance level α is the probability of:
Question 4 of 5
Power is defined as:
Question 5 of 5
At the same sample size, lowering α from 0.05 to 0.01 tends to:
