Statistics › Section J · Hypothesis testing with two samples › Topic J.4

Comparing two proportions

Does the vaccine cut infections? Does version B convert better? Comparing two proportions is the trial, the A/B test and the survey comparison in one recipe - with one new twist: under the null, the two samples share a single success rate, and the test's standard error is built from it.

Rates in competition

Clinical trials compare infection rates between vaccine and placebo groups; websites compare purchase rates between two checkout designs; surveys compare approval between two regions. The parameter is a difference of proportions, \(p_1 - p_2\); the null of no difference reads \(H_0: p_1 = p_2\); and the machinery is the familiar difference-over-standard-error shape with one genuinely new ingredient: the pooled proportion. Two of this course's biggest application areas - experimental medicine and online experimentation - run on exactly this test, so the details deserve their own topic.

The pooled proportion

Here is the subtle step. If \(H_0: p_1 = p_2\) holds, then both populations share one success rate - call it \(p\) - and the best estimate of that single rate combines both samples:

\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}

the total successes over the total trials. This pooled \(\hat{p}\) plays the role that \(p_0\) played in the one-proportion test: the null's world has one rate, and the standard error must describe that world. It is the p0-in-the-SE principle, one level up - one-proportion tests assumed a claimed value; two-proportion tests estimate the shared value from all the data.

The two-proportion z statistic

The test statistic compares the difference in sample proportions against the pooled standard error:

z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}

with \(\hat{p}\) the pooled rate from both samples. Variances still add - the SE combines the two samples' shares of blur through the \(1/n\) weights. The reference curve is the standard normal, licensed by large-enough counts.

Conditions: counts in both samples

The conditions extend the one-proportion rules to both groups. Two independent random samples - and genuinely independent of each other: different people, randomised or disjoint. The 10% rule for each sample where populations are finite. And large enough counts: at least 10 successes and 10 failures in each sample, judged with the pooled rate for a test. The count condition is what licenses the normal curve; with rare events or small groups, exact methods or a different design are the honest answer, and a statistician's early judgement call is often simply whether the counts exist to support the comparison at all.

Worked example: the vaccine trial

In a randomised trial, 15 of 200 vaccinated participants are infected over a season, against 30 of 200 unvaccinated. Do the infection rates differ? 1. \(H_0: p_1 = p_2\); \(H_a: p_1 \ne p_2\) - two-sided, as the question names no direction. 2. Pooled rate: \(\hat{p} = 45/400 = 0.1125\). 3. The pooled standard error:

\text{SE} = \sqrt{0.1125 \times 0.8875 \times \left(\frac{1}{200} + \frac{1}{200}\right)} \approx 0.0316

4. \(z = \frac{0.075 - 0.15}{0.0316} \approx -2.37\); two-sided \(p \approx 0.018\). 5. \(p \le 0.05\): reject - convincing evidence of a difference, with the vaccinated group's infection rate lower.

Worked example: the companion interval

The 95% interval for the difference does NOT use the pooled rate - a confidence interval assumes no null, so each sample supplies its own spread:

\text{SE} = \sqrt{\frac{0.075 \times 0.925}{200} + \frac{0.15 \times 0.85}{200}} \approx 0.0314

1. Estimate: \(0.075 - 0.15 = -0.075\). 2. Margin: \(1.96 \times 0.0314 \approx 0.061\). 3. Interval: \((-0.136, -0.014)\). 4. Interpretation: we are 95% confident the vaccine reduces the infection rate by between about 1.4 and 13.6 percentage points. Check: the interval excludes 0, agreeing with the test - pooled for the test, unpooled for the interval, and both consistent.

Pooled for the test, unpooled for the interval

The two-proportion case doubles the one-proportion lesson. A test lives in the null's world, so its SE uses the null's rate - here pooled from both samples, since the null claims one shared rate. An interval assumes nothing, so its SE uses each sample's own \(\hat{p}\). Mixing them up is the most common two-proportion error: a pooled interval borrows an assumption it has no right to, and an unpooled test describes a world other than the null's. The rule to keep: if the calculation assumes \(H_0\), pool; if it assumes nothing, don't. The two SEs usually differ only slightly (0.0316 against 0.0314 here), so interval and test almost always agree - but because they are not identical, a borderline case can occasionally split them.

Common mistakes to avoid

First, building the test's SE from the two separate sample rates - pool under the null. Second, building the interval's SE from the pooled rate - the interval owes the null nothing. Third, skipping the count check: 15 infections clears 10, but a rarer outcome in a smaller trial might not. Fourth, comparing groups that are not independent - the same people asked before and after is a paired design, not two samples. Fifth, quoting only the p-value: "the vaccine works (p = 0.018)" hides that the plausible reduction runs from 1.4 to 13.6 points. Sixth, letting unequal sample sizes cause panic - the SE's \(1/n\) weights handle them.

Key takeaways

Two-proportion comparisons test \(H_0: p_1 = p_2\) with \(z = (\hat{p}_1 - \hat{p}_2)/\sqrt{\hat{p}(1-\hat{p})(1/n_1 + 1/n_2)}\), where \(\hat{p}\) pools both samples' successes - the null's single shared rate, estimated from everything. Conditions: independent random samples, the 10% rule, and 10 successes-and-failures per sample. The vaccine trial: pooled \(\hat{p} = 0.1125\), SE 0.0316, \(z \approx -2.37\), \(p \approx 0.018\) - reject; the unpooled interval runs from \(-0.136\) to \(-0.014\), a 1.4-to-13.6-point reduction. Pooled for tests, unpooled for intervals - and always report the size.

Flashcards

Flashcard

H0: p1 = p2

The no-difference null for two rates

One shared rate in the null's world.

Test your understanding

Question 1 of 5

A trial has 15 infections among 200 vaccinated and 30 among 200 unvaccinated. What is the pooled proportion?

Question 2 of 5

With pooled p̂ = 0.1125 and n1 = n2 = 200, the test's standard error is about:

Question 3 of 5

With p̂1 = 0.075, p̂2 = 0.15 and SE ≈ 0.0316, the test statistic is about:

Question 4 of 5

In the TEST's standard error, which success rate appears?

Question 5 of 5

The 95% interval for the difference is (−0.136, −0.014). At α = 0.05, the test's decision will be:

← Previous topicNext topic →