American Government › Section F · Public opinion › Topic F.2
How public opinion is measured
A well-designed poll of about 1,000 people can describe the views of more than 250 million adults, while a badly designed poll of millions can miss by a mile. The difference lies in sampling, question design and honest reporting of uncertainty.
From straw polls to science
Newspapers were counting opinion as early as 1824, when a Pennsylvania paper reported an informal tally of presidential preferences. Such straw polls collect views from whoever happens to respond, so they show only what that group thinks. Many online polls on news sites and social media are straw polls today. A scientific poll is different: it defines the population it wants to describe, selects respondents by a method that gives every member a known chance of inclusion, asks carefully tested questions and reports its margin of error.
The lesson of 1936
The Literary Digest magazine had correctly called several presidential elections by mailing millions of mock ballots. In 1936 it sent about 10 million, mainly to its readers and to people listed as owning cars or telephones, and about 2.4 million came back. It predicted that the Republican Alf Landon would win about 57 percent of the vote. Franklin Roosevelt won with about 61 percent and every state except Maine and Vermont. George Gallup, using a far smaller sample of about 50,000 chosen to resemble the electorate, correctly predicted a Roosevelt victory. The episode made sampling method, not sheer size, the test of a poll.
Worked example: diagnosing the Digest's failure
Identify what went wrong. 1. Sampling frame: the mailing list came from magazine subscriptions and car and telephone registrations. In the Depression those groups were richer than average and leaned Republican. 2. Nonresponse: only about a quarter of recipients replied, and later research suggests that people who disliked Roosevelt were especially likely to send ballots back. 3. Size did not help: 2.4 million biased answers still gave a biased result. 4. Gallup's advantage: a smaller sample built to match the electorate reduced both problems. 5. Conclusion: a sample must represent the population; a large unrepresentative sample only measures the wrong people more precisely.
Populations and samples
Pollsters first define the population: all adults, registered voters or likely voters in one state, for example. Election polls often screen for likely voters, using questions about past voting and interest, because many adults will not vote. A random sample gives each member of the population a known, usually equal, chance of selection, so the sample's views mirror the population's within a calculable range. A representative sample matches the population on characteristics such as age, sex, race, education and region. Random selection produces representativeness on average, and pollsters adjust the rest by weighting, giving underrepresented groups more weight in the final figures.
Sample size and margin of error
Because a sample is not the whole population, every poll carries sampling error. The margin of error states how far the result is likely to be from the true figure, usually with 95 percent confidence. For a simple random sample of \(n\) people and a result near 50 percent, a handy approximation is
For 1,000 respondents that gives about 3 percentage points. Quadrupling the sample only halves the margin, so pollsters rarely go far beyond 1,000 to 1,500. Subgroups, such as voters under 30, have smaller samples and therefore larger margins. Weighting also widens the true margin slightly.
Worked example: is the race really close?
A poll of 1,000 likely voters finds Candidate A on 48 percent and Candidate B on 45 percent, with a margin of error of 3.1 points. 1. Check the margin: \(1.96 \times \sqrt{0.25/1000} \approx 0.031\), or 3.1 points. 2. Range for A: 48 minus 3.1 to 48 plus 3.1, about 44.9 to 51.1 percent. 3. Range for B: about 41.9 to 48.1 percent. 4. The ranges overlap, and the uncertainty on the gap between two candidates is nearly twice the stated margin. 5. Conclusion: A's 3-point lead is within the margin of error, so the race is too close to call from this poll alone. Several polls showing the same lead would be stronger evidence.
How polls reach people
Early polls interviewed people face to face, which is accurate but slow and expensive; exit polls and some academic surveys still do it. Telephone polling grew from the 1970s, using random-digit dialling to reach unlisted numbers. As people abandoned landlines and stopped answering unknown callers, response rates fell sharply, and polls had to call mobile phones by hand, because federal law restricts automated dialling to them. Many pollsters now use online panels, text messages or a mix of methods. The best panels recruit members randomly, for example by postal address; opt-in panels, where anyone can sign up, are cheaper but need heavier weighting and can be less reliable. Campaigns also commission benchmark polls at the start of a race and tracking polls that follow opinion day by day.
Wording and order
Small changes in wording can move results. For decades the General Social Survey has found much more support for spending on "assistance to the poor" than on "welfare", though the programmes overlap. A leading question pushes respondents towards an answer: "Given rising crime, do you support more police funding?" invites agreement that a neutral version would not. Question order matters too, because earlier questions can prime later ones. Offering an explicit "don't know" option reduces nonattitudes, answers given by people who have no real opinion but do not want to admit it. Good pollsters test questions and publish their exact wording.
When respondents and timing mislead
Some errors come from respondents. Social desirability bias leads people to give the answer they think is acceptable, which is why surveys overstate turnout. The Bradley effect, named after the Black Los Angeles mayor Tom Bradley, who led in polls but narrowly lost the 1982 race for governor of California, is the theory that some white voters hid their opposition to a Black candidate; researchers dispute how often it occurs. Timing matters as well: in 1948 major pollsters stopped polling weeks before election day and missed Harry Truman's late surge. In 2016 many state polls underestimated Donald Trump, partly because they did not weight by education.
Push polls and judging quality
A push poll is not research at all: it is campaign persuasion disguised as a survey, asking questions such as "Would you still support Smith if you knew she had raised taxes?" to spread negative claims. Professional bodies such as the American Association for Public Opinion Research condemn the practice. To judge a real poll, look for who paid for it and who conducted it, the population and how respondents were selected, the sample size and margin of error, the dates of fieldwork, and the exact question wording. A poll that hides these details deserves suspicion, however striking its findings.
Common mistakes to avoid
First, trusting a big sample over a well-chosen one; the Literary Digest had millions of responses and still failed. Second, reading a lead smaller than the margin of error as a clear win. Third, forgetting that subgroup results carry larger margins than the whole sample. Fourth, treating an online straw poll as scientific. Fifth, confusing a push poll, which is campaign messaging, with a leading question in a genuine survey. Sixth, assuming polls predict the future; they describe opinion at the time of fieldwork, and opinion can change.
Key takeaways
A scientific poll defines a population, draws a random and representative sample, asks neutral questions and reports a margin of error. The 1936 Literary Digest failure showed that sampling method matters more than size. A sample of about 1,000 gives a margin of roughly 3 points at 95 percent confidence, and subgroups have wider margins. Falling response rates have pushed pollsters towards mobile, online and mixed methods with weighting, especially by education. Wording, order, social desirability and timing can all bias results. Push polls are persuasion, not research.
Flashcards
Flashcard
An informal poll of whoever chooses to respond
Test your understanding
Question 1 of 5
Why did the Literary Digest's 1936 poll fail despite about 2.4 million responses?
Question 2 of 5
Roughly what margin of error does a simple random sample of 400 people have for a result near 50 percent?
Question 3 of 5
A poll shows Candidate A on 47 percent and Candidate B on 45 percent, with a margin of error of 3 points. What is the best conclusion?
Question 4 of 5
A caller asks, "Would you still vote for Jones if you knew he had missed half his votes?" and ends without recording demographics. What is this?
Question 5 of 5
Why do surveys often report higher turnout than actually occurred?
