Statistics › Section A · Sampling and data › Topic A.5

Experimental design and ethics

Does the programme work? A well-designed experiment is the clearest way to find out. This topic covers the vocabulary of experiments, the design principles that turn data into evidence, and the ethical rules that keep that evidence honest.

What experiments are for

Does a new reading app raise test scores? Does extra sleep improve reaction time? Questions like these are answered with randomised experiments, and proper design is what makes the answers reliable. An experiment deliberately imposes a treatment and then measures the response; a study that merely observes cannot isolate the cause. This topic builds the vocabulary of experiments - explanatory and response variables, treatments, control groups - the design principles that make results trustworthy, and the ethical rules every study must follow.

Observational studies versus experiments

An observational study measures variables without intervening: it records who eats breakfast, who exercises, who tutors. An experiment imposes a treatment and records what happens. The difference matters because observational groups are self-selected: students who eat breakfast may also sleep more, keep steadier routines or have more support at home. Seeing that breakfast eaters attend school more often is real information, but it shows association, not causation. A randomised experiment, where the researcher assigns the treatment, is the direct way to isolate the explanatory variable and support a cause-and-effect claim.

Explanatory variables, response variables, treatments

Every experiment investigates the relationship between two variables. The explanatory variable is what the researcher changes or imposes - the suspected cause. The response variable is what gets measured - the suspected effect. Each value the explanatory variable takes is a treatment. In a study of a tutoring programme, the explanatory variable is the study method, the treatments are "new programme" and "standard practice", and the response variable is the score on a common exam. Naming these three pieces is the first step in writing down any design.

Lurking variables and confounding

A lurking variable is a factor not part of the study that influences both the explanatory and the response variable - regular exercise lurking behind a vitamin supplement, motivation lurking behind front-row seats. When two factors' effects cannot be separated, they are confounded: a student who both studies more and sits in a favourite seat earns a high score, and no one can say which action deserves the credit. Confounding is why observational studies hesitate at causation: without design, the explanation you like and the one you never measured are tangled together.

Random assignment

The cure for lurking variables is to let chance decide treatments. Random assignment uses a chance process - coin flips, random numbers - to allocate subjects to treatment groups. Because the assignment ignores everything about the subjects, lurking variables such as age, health, motivation and sleep tend to spread roughly evenly across the groups. After that, the only systematic difference between the groups is the treatment itself, so a difference in the response larger than chance alone would produce can be traced back to it. Note the contrast with random sampling: sampling decides who enters the study, assignment decides who gets which treatment.

Control and replication

Two more principles complete the design. Control: give one group the standard treatment or a placebo so the effect has something to be compared against - without a baseline, "improvement" means nothing. Replicate: use enough subjects, because chance differences between individuals average out in larger groups. A tutoring study with three students per group tells you almost nothing, since one unusual student swings the result; the same design with 100 per group speaks clearly. Both principles work alongside random assignment: assign fairly, compare against a control, and repeat enough times for the pattern to rise above the noise.

Blinding and the placebo effect

Expectation alone changes outcomes - the placebo effect. Participants who believe they received the active treatment often improve, and evaluators who know who received it may judge them more favourably. Blinding removes that knowledge. In a single-blind study, subjects do not know which treatment they got; in a double-blind study, the people measuring the response do not know either. A placebo is the control group's fake treatment - a sugar pill or sham session identical in every way except the active ingredient. Blinding is not always possible (a student knows whether she joined the tutoring programme), but often the evaluators can still be kept in the dark.

Worked example: designing a tutoring experiment

A school wants to know whether a new tutoring programme improves maths scores. 1. Explanatory variable: the tutoring programme (offered or not). 2. Treatments: new programme; standard practice. 3. Response variable: score on the same end-of-term exam. 4. Recruit 200 student volunteers and randomly assign 100 to each treatment - this spreads prior ability and motivation roughly evenly. 5. Keep the exam graders blind to group membership. 6. Check the design: subjects cannot be blinded here, but random assignment plus blinded grading covers the main threats, so a large score difference points to the programme itself.

Worked example: when an experiment is impossible

Can growing up with pets reduce children's allergies? 1. Explanatory variable: pets in the home (yes or no). 2. Could we assign it? No - a researcher cannot randomly place pets in half the families and forbid them in the rest for years on end. 3. The study must therefore be observational: compare allergy rates in children from homes with and without pets. 4. Conclusion allowed: association only. Families with pets may differ in where they live, income or hygiene habits, and any of those could drive allergy rates. 5. Check: the barrier here is that the explanatory variable cannot practically or ethically be assigned.

Ethics: honest data and fair treatment

Statistics earns a bad name when its numbers are bent. Never acceptable: fabricating results, altering data after collection, quietly dropping inconvenient cases, changing the measuring instrument without reporting it, or inventing responses - and copying a real participant's answers into empty slots is still fraud. Softer failures also mislead: surveying only a comfortable neighbourhood, or not returning to houses where nobody was home, biases the sample quietly. Human participants additionally deserve informed consent, protection from harm, and confidentiality of their data; in the US, an institutional review board (IRB) reviews such studies before they begin. And any funded study deserves one more question: who paid for it, and what did they want the answer to be?

Common mistakes to avoid

First, concluding cause and effect from a single observational study - random assignment is what justifies that step. Second, confusing random sampling with random assignment: one selects the participants, the other allocates the treatments. Third, treating the placebo as "no treatment": it is a deliberate control that matches expectations. Fourth, assuming blinding is all-or-nothing: participants may be unblindable while evaluators stay blind. Fifth, forgetting replication: with tiny groups, chance swings dominate. Sixth, assuming a published study is an honest one - check who funded it and whether the methods were reported in full.

Key takeaways

Experiments impose treatments and measure responses; observational studies only watch, so they show association, never guaranteed causation. Name the explanatory variable, the treatments and the response variable before designing anything. Lurking variables confound until random assignment spreads them roughly evenly across groups - the step that earns a cause-and-effect conclusion. Complete the design with a control group, a placebo where possible, blinding of subjects and evaluators, and enough subjects for real effects to surface. And keep the enterprise honest: real data, reported methods, consent and confidentiality.

Flashcards

Flashcard

Experiment

Imposes a treatment and measures the response

Random assignment is what supports cause and effect.

Test your understanding

Question 1 of 5

A study records that students who eat breakfast have better attendance, but no one assigned students to eat breakfast. The study can conclude:

Question 2 of 5

In a tutoring experiment, the score on the common end-of-term exam is:

Question 3 of 5

Randomly assigning subjects to treatment groups mainly:

Question 4 of 5

Why does the control group receive a placebo?

Question 5 of 5

A researcher randomly selects 200 students for a study, then uses chance to decide who joins the tutoring programme. Which random act decides who gets the programme?

← Previous topicNext topic →