Statistics › Section A · Sampling and data › Topic A.2

Data, sampling, and variation

Data are only as good as the way they were gathered. This topic covers the standard sampling designs, the ways samples go wrong, and why two fair samples rarely agree exactly.

Where data come from

Recall that a population is the whole group a study targets and a sample is the part actually measured; we write \(N\) for the population's size and \(n\) for the sample's size. A census measures everyone, but most studies sample: crash-testing every car off a line would destroy the product, and interviewing millions costs too much. The data a sample yields can be qualitative (categorical, like blood type or favourite sport) or quantitative, and quantitative data are discrete when counted and continuous when measured. That vocabulary decides which sampling designs and summaries make sense, so name the kind of data before collecting anything.

Sampling variability

Take two random samples from the same population and you will almost certainly get different results. One survey of 50 households might find a different recycling rate than another, even though both were drawn fairly. Neither researcher made a mistake: this natural disagreement is sampling variability, and the gap between a sample's result and the population's true value is called sampling error. It comes from sampling itself, and larger samples tend to shrink it. Nonsampling errors have other causes, such as a faulty counter or a mis-keyed answer, and a bigger sample does not fix them. So a difference between two random samples is not, by itself, evidence that either method was biased.

Simple random samples

A simple random sample (SRS) gives every possible sample of size \(n\) the same chance of selection: every group of \(n\) individuals is equally likely. That requirement is stronger than giving every individual the same chance, which would still allow lopsided samples such as everyone from one classroom. In practice an SRS is drawn with a random number generator or by shuffling the whole list, and it removes the researcher's influence over who gets picked. Surveys usually sample without replacement, so no one is picked twice; sampling with replacement returns each pick to the pool. When the population is far larger than the sample, the two give almost the same results. Because selection is chance-based, we can compute how much samples are likely to vary, the foundation of later inference.

Stratified, cluster, and systematic designs

Three other random designs suit different situations. To take a stratified sample, divide the population into strata (year groups, regions, age bands) and draw a simple random sample within each, so every group is represented, often in proportion to its size. To take a cluster sample, divide the population into clusters (schools, stores, city blocks), randomly select a few clusters, and include everyone in them; this is cheaper when a population is spread out, but a cluster sample often gives less precise estimates than an SRS of the same size. A systematic sample picks a starting point at random and then takes every kth individual on a list, and is only fair when the list has no hidden pattern.

Convenience and voluntary response samples

The easiest samples to take are the least trustworthy. A convenience sample takes whoever is easy to reach: shoppers at one mall, your friends, the first 20 email replies. A voluntary response sample lets people choose whether to respond, as in call-in votes, comment sections and online polls. Both favour some outcomes over others in a systematic way: friends share your habits, one mall misses everyone who shops elsewhere, and people who bother to vote online tend to hold the strongest opinions. That systematic favouring is called bias. Convenience and voluntary samples can still be useful for a quick pilot or for generating ideas, but they cannot support claims about the whole population.

Bias inside a well-run sample

Even a random design can be spoiled at the edges. Undercoverage happens when the sampling frame - the list you draw from - misses part of the population, like a survey that dials only landlines and so misses mobile-only households. Nonresponse happens when selected individuals cannot be reached or refuse to answer, and nonresponders often differ from responders. Response bias arises from the question itself: leading wording, sensitive topics, or the wish to look good can distort answers. Some studies also deserve a look at who funded them, since self-interest can shape what is asked and how results are reported. Checking these points is part of reading any statistical claim critically.

Worked example: naming the design

Identify the design in each plan. a) A school district numbers all 2,400 students and a computer draws 200 of them: a simple random sample. b) A principal surveys a random 10% of each year group, so every year is represented: a stratified sample. c) A researcher randomly chooses four homerooms and surveys every student in them: a cluster sample. d) An inspector samples every 40th crate on the day's list after a random first draw: a systematic sample. e) A student asks friends in the cafeteria: a convenience sample. Check: in each case ask what the chance mechanism selects - individuals, a slice of every group, whole groups, positions on a list, or nobody in particular.

Worked example: two samples, two answers

Two students each survey 50 randomly chosen households about recycling. Maria finds 31 households that recycle: \(31/50 = 0.62\). Jules finds 36: \(36/50 = 0.72\). 1. Both samples are random, so both are fair. 2. The gap, \(0.72 - 0.62 = 0.10\), is sampling variability at work. 3. Neither 0.62 nor 0.72 is wrong; both estimate the same population proportion, and later topics explain how far such estimates typically land from it. 4. Check: with 50 households, one household shifts a proportion by \(1/50 = 0.02\), so a five-household difference fully accounts for the 0.10 gap. Small samples swing a lot; that is variability, not bias.

A famous failure: the 1936 Literary Digest poll

In 1936 the magazine Literary Digest mailed about 10 million straw-vote ballots, drawn mostly from phone directories, car registrations and club membership lists, and about 2.4 million came back. The poll predicted a comfortable victory for the Republican candidate, Alf Landon; Franklin Roosevelt instead won easily, taking about 61% of the popular vote. The failure was the method, not the size: in the Depression, phone and car owners were wealthier and more Republican than the average voter, and the people who mailed ballots back chose themselves - voluntary response on top of undercoverage. George Gallup, with a far smaller quota sample of about 50,000 chosen to mirror the electorate, called the winner correctly. No sample size can rescue a biased design.

Common mistakes to avoid

First, believing a huge sample guarantees accuracy - size reduces sampling variability but does nothing for bias, as the Literary Digest showed. Second, treating a voluntary online vote as representative of anything beyond its own voters. Third, confusing stratified with cluster sampling: strata are sampled within, clusters are sampled whole. Fourth, starting a systematic sample at a convenient place rather than a random one, which removes the chance element. Fifth, calling a haphazard process "random": grabbing whoever walks past is not chance. Sixth, expecting two good studies to agree exactly, and concluding that one must be flawed whenever they differ.

Key takeaways

Data come from populations (size \(N\)) through samples (size \(n\)), and random samples naturally differ - that is sampling variability, not a mistake; a bigger sample shrinks sampling error but not nonsampling error. A simple random sample makes every group of \(n\) individuals equally likely; stratified samples draw within every stratum, cluster samples take whole groups, and systematic samples take every kth individual from a random start. Convenience and voluntary response sampling invite bias, and so do undercoverage, nonresponse, response bias and self-interest, even inside otherwise random designs. Judge a study by its method: the 1936 Literary Digest poll failed with about 2.4 million responses, while Gallup's far smaller sample named the right winner.

Flashcards

Flashcard

Sampling variability

The natural tendency of random samples to give different results

A difference between samples is not a mistake.

Test your understanding

Question 1 of 5

A quality inspector picks a starting crate at random and then checks every 40th crate after it on the day's list. This is a:

Question 2 of 5

A researcher randomly selects four homerooms at a school and surveys every student in them. This is a:

Question 3 of 5

Why did the 1936 Literary Digest poll predict the wrong winner?

Question 4 of 5

Two students each take random samples of 50 households and get different recycling rates. The best explanation is:

Question 5 of 5

A TV station's online poll receives 80,000 votes and reports that 79% said yes. The main problem is:

← Previous topicNext topic →