Statistics › Section N · Counting, Bayesian inference, and multiple regression › Topic N.2
Bayes' theorem as learning from evidence
Swapping P(positive | disease) for P(disease | positive) is one of the commonest errors in probability. Bayes' theorem is the correct way to make that reversal: a rule for turning evidence into updated belief, one observation at a time.
The reversal the fallacy needs
Tests, witnesses and sensors all offer probabilities of the form P(evidence | state) - the chance of the signal GIVEN the condition. Decisions need the reversed quantity: P(state | evidence) - the chance of the condition given the signal. Treating one as the other is the confusion of the inverse; in court it is called the prosecutor's fallacy, when the chance of the evidence given innocence is mistaken for the chance of innocence given the evidence. Bayes' theorem is the honest reversal, and the Bayesian framework built on it treats learning itself as a calculation: beliefs go in, evidence arrives, updated beliefs come out.
Prior, likelihood, posterior
A Bayesian analysis names three objects. The PRIOR: what was believed before the evidence - a prevalence rate, a base rate, past studies. The LIKELIHOOD: how probable the observed evidence is under each hypothesis - a test's sensitivity and false-positive rate. The POSTERIOR: the updated belief after the evidence, formed by combining them. The recipe, Bayes' theorem in its working form:
The constant that restores exact probabilities comes from ensuring the total is 1, but proportions suffice for thinking.
Bayes' theorem, exactly
For two hypotheses \(A\) and \(B\) and evidence \(E\):
The numerator is the evidence's probability under \(A\), weighted by \(A\)'s prior; the denominator runs the same product over every hypothesis and sums - it is P(E), the evidence's total probability. The formula's shape explains the fallacy: the reversed quantity depends not just on how loudly evidence shouts for \(A\) (the likelihood) but on how common \(A\) was to begin with (the prior). A rare condition can stay improbable even after one positive test, which is where intuition usually goes wrong.
Worked example: the medical test, reversed properly
A disease has 1% prevalence; a test detects it 95% of the time when present and falsely signals 5% of the time when absent. A patient tests positive. 1. Prior: \(P(D) = 0.01\). 2. Likelihoods: \(P(+ \mid D) = 0.95\), \(P(+ \mid \text{not } D) = 0.05\). 3. \(P(D \mid +) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.05 \times 0.99} = \frac{0.0095}{0.059} \approx 0.16\). 4. Only 16% - most positives are false, because the disease is rare. Check: imagine 10,000 patients - 100 diseased (95 true positives) against 9,900 healthy (495 false positives); \(95/590 \approx 0.16\), the frequency picture confirming the formula.
Worked example: evidence accumulates
The same patient tests positive again on an independent test (same 95% and 5%). 1. Today's posterior is tomorrow's prior: \(P(D) = 0.16\) now. 2. Update: \(P(D \mid ++\,) = \frac{0.95 \times 0.16}{0.95 \times 0.16 + 0.05 \times 0.84} = \frac{0.152}{0.194} \approx 0.78\). 3. Two positive tests raise belief from 16% to about 78% - evidence piling up exactly as the framework promises. 4. Check: the odds form gives the same result in one line, below - and a third positive would clear 95%. Bayesian updating turns a stream of signals into steadily sharpening belief.
The odds form
Bayes' theorem is fastest in odds:
The likelihood ratio compares how much more probable the evidence is under one hypothesis than the other. The medical test's ratio: \(0.95/0.05 = 19\) - a positive test is 19 times likelier with the disease. Prior odds 1 to 99, times 19, gives posterior odds 19 to 99: a probability of \(19/118 \approx 0.16\) - the same 16%, one line. The second positive multiplies the odds by 19 again: \(19 \times 19 = 361\) to 99, a probability of \(361/460 \approx 0.78\) - exactly the 78% from the long route, with no intermediate rounding. Mental updating, exact and fast.
Choosing priors honestly
The prior is where Bayesian analysis is honest about its own inputs. Informative priors come from real knowledge - past studies, established base rates, expert calibration. Weakly informative priors spread belief broadly when little is known, letting the data speak. The discipline is openness: state the prior, justify it, and let readers judge. With plenty of data the likelihood dominates and any reasonable prior reaches a similar posterior; with little data the prior shapes the conclusion - which is precisely when hiding it would be most misleading. Choosing a prior AFTER seeing the data, to steer the result, is the Bayesian version of p-hacking.
Sequential updating everywhere
Today's posterior is tomorrow's prior - and that recursion powers modern systems. Spam filters update each word's spam-probability with every message marked junk. Navigation systems fuse each new sensor reading with the current position estimate. Diagnostic systems accumulate symptoms one at a time, each revising the differential. The same loop - prior, evidence, posterior, repeat - runs wherever machines learn from streams of data, which is one reason the Bayesian framework has spread far beyond its statistical origins.
Common mistakes to avoid
First, ignoring the base rate - the fallacy Bayes' theorem exists to prevent; a test with 95% sensitivity and 5% false positives, used on a 1% disease, still leaves 84% of positives healthy. Second, choosing the prior after the data to manufacture a conclusion. Third, confusing likelihood with posterior: P(+ | D) is 95%, P(D | +) is 16%, two different questions with very different answers. Fourth, forgetting that a strong prior with weak data produces conclusions that largely echo the prior - the posterior is a blend, not a fresh start. Fifth, double-using evidence already absorbed into the prior. Sixth, treating Bayes' theorem as optional when the question is literally a reversal.
Key takeaways
Bayes' theorem reverses conditional probabilities: \(P(A \mid E)\) combines how loudly the evidence speaks (the likelihood) with how common the hypothesis was (the prior). Posterior \(\propto\) likelihood \(\times\) prior; in odds, posterior odds = prior odds \(\times\) likelihood ratio. The medical test: 1% prevalence, 95%/5% test - a positive gives \(0.0095/0.059 \approx 16\)%, odds form 19-to-99 in one line; a second positive lifts belief to about 78% - today's posterior is tomorrow's prior, the recursion behind spam filters, navigation and every learning-from-streams system. Priors should be stated and justified: with data they fade, without data they rule.
Flashcards
Flashcard
The reversed probability decisions need
Test your understanding
Question 1 of 5
In Bayesian inference, the posterior is proportional to:
Question 2 of 5
A disease has 1% prevalence; a test is 95% sensitive with a 5% false-positive rate. Given ONE positive result, the probability of disease is about:
Question 3 of 5
The same patient then tests positive again on an independent test. Belief in disease rises to about:
Question 4 of 5
In the odds form of Bayes' theorem, posterior odds equal:
Question 5 of 5
The medical test's likelihood ratio for a positive result is:
