Statistics › Section L · Linear regression and correlation › Topic L.6

Residuals, outliers, and influential points

A regression line is only useful if it fits - and the residuals, the leftovers the line could not explain, are how that judgement is made. One plot tells more than any summary number, and two numbers say how much the line is worth.

Leftovers tell the truth

A fitted line makes a prediction for every point in the scatterplot, and every point misses by some amount. The residual is observed minus predicted:

e = y - \hat{y}

Positive residuals sit above the line (the model underpredicted); negative, below. Least squares guarantees the residuals sum to zero - over- and under-predictions balance - and the residuals carry all the variation the line failed to explain. Reading them is how the model is audited.

The residual plot

Plot the residuals against \(x\) (or against the predicted values). A good linear model leaves a formless horizontal band around zero - random scatter, no pattern, because any remaining pattern is structure the model missed. Two patterns matter most: a CURVE means the relationship is not linear - a straight line is the wrong tool; a FAN means the spread grows with \(x\) - predictions get less reliable where the spread is widest. Residual plots reveal problems the original scatterplot hides, because they remove the line's own slope from view and leave only its failings.

r²: the fraction explained

The coefficient of determination \(r^2\) measures the fit in one number: the fraction of the variation in \(y\) explained by the linear relationship with \(x\). If \(r = 0.8\) between house size and price, then \(r^2 = 0.64\): 64% of price variation is explained by size, and 36% belongs to other factors. \(r^2\) runs from 0 (the line explains nothing) to 1 (perfect), and is often reported as a percentage. It compares the residuals' squared misses with the total variation about \(\bar{y}\):

r^2 = 1 - \frac{\sum e^2}{\sum (y - \bar{y})^2}

It is best quoted alongside its companion \(s_e\).

The residual standard deviation: the typical miss

The residual standard deviation \(s_e\) gives the typical size of prediction errors in \(y\)'s own units: if \(s_e = 36\) thousand dollars, a typical house prediction misses by about $36,000. Where \(r^2\) says what share of the story the line tells, \(s_e\) says how wrong a prediction usually is - and for decisions, the second is often more useful. Report the pair: "64% explained, typical miss $36,000" says what "\(r^2 = 0.64\)" alone cannot.

Outliers and influential points

An outlier in regression has a large residual - it sits far from the line the rest of the data chose. An influential point is more dangerous: remove it and the fitted slope changes substantially, usually because it has an extreme \(x\)-value (high leverage), giving it disproportionate pull on the line's tilt. The discipline: fit the model with and without any suspicious point, report both if the verdict changes, and investigate why the point is unusual. Deleting inconvenient points without cause is data-dredging; staying silent while one observation decides the slope is no better.

Transformations when the line is wrong

A curved residual plot is a diagnosis, not a defeat: transform a variable and refit. Exponential growth (population, compound interest, viral spread) becomes linear by taking \(\ln y\) and regressing on \(x\); power relationships become linear by taking logs of both variables. Then predictions transform back to the original units. Many curved relationships are straight lines in disguise, once the right logarithm is taken.

Worked example: the fuel-efficiency line

A line predicting fuel efficiency from car weight reports \(r^2 = 0.75\) and \(s_e = 2.1\) mpg - respectable numbers, typical miss about 2 mpg. But the residual plot curves: residuals run positive at both ends and negative in the middle - the line systematically underpredicts the lightest and heaviest cars while overpredicting the midrange. Verdict: despite the decent \(r^2\), the linear model is the wrong shape; a transformation of weight (or a curved model) would fit better. Check: this is the lesson of this whole topic - the summaries said "fine", and the residuals said "wrong".

Worked example: reading the pair of numbers

The study-time example (five students, hours 1 to 5, scores 2 to 5) had \(r = 0.775\), so \(r^2 \approx 0.60\): about 60% of the score variation is explained by study time, 40% by everything else - talent, sleep, test anxiety. Its line \(\hat{y} = 2.2 + 0.6x\) leaves residuals \(-0.8, 0.6, 1.0, -0.6, -0.2\), so \(\sum e^2 = 2.4\) and, dividing by \(n - 2 = 3\), \(s_e = \sqrt{2.4/3} \approx 0.89\) points: a typical prediction misses by about one point. Report: "study time explains about 60% of score variation, and predictions are typically within about a point" - adding that five points are too few to judge the residual plot's shape. Check: \(r^2\) without \(s_e\) hides the practical size; \(s_e\) without \(r^2\) hides the model's reach. The pair, plus the residual plot, is the full report.

Common mistakes to avoid

First, judging fit by \(r^2\) alone - the fuel example's 0.75 hid a curve. Second, skipping the residual plot entirely. Third, treating a high \(r^2\) as causation: 75% explained by weight is 75% associated with weight. Fourth, deleting outliers without justification, or keeping silent about influential points that decide the slope. Fifth, comparing \(r^2\) across models with different response variables - different \(y\)'s explain different denominators. Sixth, extrapolating from a model whose residuals show the line was wrong even inside the data.

Key takeaways

Residuals are observed minus predicted; they sum to zero and carry the unexplained variation. The residual plot is the audit: random scatter approves, a curve indicts the straight line, a fan indicts constant-spread claims. \(r^2\) is the fraction of \(y\)-variation the line explains (\(r = 0.8\) gives 64%; \(r = 0.9\) gives 81%); \(s_e\) is the typical miss in \(y\)-units - report the pair. Outliers have large residuals; influential points move the slope - fit with and without. Curved residuals call for transformations (logs for growth and power laws). The fuel line's 0.75 hid a curve; the residuals told the truth the summaries buried.

Flashcards

Flashcard

e = y − ŷ

Residual: observed minus predicted

The line's miss at each point.

Test your understanding

Question 1 of 5

If r = 0.9 between study time and score, what percent of the variation in scores does study time explain?

Question 2 of 5

A residual plot shows a smooth curve, positive at both ends and negative in the middle. The verdict:

Question 3 of 5

Point A has a large residual but a typical x-value. Point B has an extreme x-value and sits off the trend of the other points. Removing which is more likely to change the slope substantially?

Question 4 of 5

A regression reports r² = 0.75 and s_e = 2.1 mpg. The typical prediction misses by about:

Question 5 of 5

Exponential growth data (like compound interest) become linear for regression by:

← Previous topicNext topic →