Statistics › Section L · Linear regression and correlation › Topic L.3

The least-squares regression line

A scatterplot shows the relationship; a regression line summarises it into a prediction machine. The least-squares line is the one that misses the data least - and its slope and intercept carry the story in usable units.

From scatter to line

When a scatterplot shows a roughly linear relationship, one line can summarise it and predict the response from the explanatory variable: \(\hat{y} = b_0 + b_1x\). The slope \(b_1\) is the predicted change in \(y\) for a one-unit increase in \(x\); the intercept \(b_0\) is the predicted \(y\) at \(x = 0\), meaningful only when zero is a real place in the story. The hat on \(\hat{y}\) marks a prediction, not an observation - the line's opinion, never the data's fact.

Least squares: the criterion

Many lines pass through a scatterplot; which one deserves to represent it? Each point's vertical miss is its residual, \(y - \hat{y}\): observed minus predicted, positive for a point above the line and negative below. The least-squares criterion picks the line that makes the sum of SQUARED residuals as small as possible. Squaring makes every miss count positively and penalises large misses heavily, so one point far from the pattern can pull the line towards itself. The result is the line of best fit in this specific sense, and it is the line calculators and software report.

The formulas

The least-squares slope ties directly to the correlation topic:

b_1 = r\,\frac{s_y}{s_x}

and the line always passes through the point of means \((\bar{x}, \bar{y})\), so

b_0 = \bar{y} - b_1\bar{x}

The slope formula says two things at once: stronger correlation, steeper standardised slope; and the \(s_y/s_x\) factor converts the unitless \(r\) back into \(y\)-units per \(x\)-unit. The mean-point property is a free check on every calculation - the fitted line must pass through \((\bar{x}, \bar{y})\).

Worked example: the house-price line

For a set of houses: size \(x\) in hundreds of square feet, \(\bar{x} = 20\), \(s_x = 5\); price \(y\) in thousands of dollars, \(\bar{y} = 300\), \(s_y = 60\); correlation \(r = 0.8\). 1. Slope: \(b_1 = 0.8 \times \frac{60}{5} = 9.6\) thousand dollars per hundred square feet - about $9,600 per extra 100 square feet. 2. Intercept: \(b_0 = 300 - 9.6 \times 20 = 300 - 192 = 108\). 3. Line: \(\hat{y} = 108 + 9.6x\). 4. Check: at \(x = 20\), \(\hat{y} = 108 + 192 = 300\) - the line passes through the means, as it must.

Worked example: using and misusing the line

The house sizes run from 15 to 25 (hundreds of square feet). 1. Prediction inside the range: a 24-hundred-square-foot house (2,400 sq ft) is predicted at \(108 + 9.6 \times 24 = 338.4\) thousand dollars - a defensible interpolation. 2. Extrapolation: a 40-hundred (4,000 sq ft) mansion predicts \(108 + 384 = 492\) - but the line was fitted to mid-size houses, and mansions obey their own market: the prediction is arithmetic without support. 3. Check: the fitted line's intercept, 108 thousand for a zero-size house, is equally meaningless - no house has zero size. The line speaks only where the data lived.

Interpreting slope and intercept

Interpretations carry units or fail. Slope: "each additional hundred square feet adds about $9,600 to the PREDICTED price" - the word predicted matters twice, keeping causation out and the model's opinion in. Intercept: 108 thousand dollars at zero size is not meaningful, since no house has zero size; interpret an intercept only when \(x = 0\) is realistic - a machine's startup cost, a plan's base fee. And in observational data a slope describes association: houses with more space sell for more, and the line never says why.

Regression toward the mean

Because \(|r| < 1\), predictions shrink in standardised terms: an individual \(k\) standard deviations above average in \(x\) is predicted only \(rk\) standard deviations above average in \(y\). A house 2 SDs above mean size sits at \(x = 30\), predicted \(1.6\) SDs above mean price: \(300 + 1.6 \times 60 = 396\) - and the line agrees (\(108 + 9.6 \times 30 = 396\)). Very tall parents tend to have tall children, but on average less extreme ones; a standout first season is often followed by a more ordinary one. Francis Galton described this shrinkage in parents' and children's heights in the 1880s, and the technique's name, regression, comes from it.

Regression is not symmetric

The line predicting \(y\) from \(x\) is not the line predicting \(x\) from \(y\) unless \(r = \pm 1\). The choice of response - which variable needs predicting - belongs on the \(y\)-axis, and it is a decision about the question, not the data: predicting price from size fits a buyer's question; predicting size from price fits a marketing one. The two lines disagree because least squares minimises vertical misses, and "vertical" depends on which variable responds.

Common mistakes to avoid

First, extrapolating far beyond the data's range - the 4,000-square-foot prediction is arithmetic, not evidence. Second, reading the slope as a causal effect from observational data: association, not mechanism. Third, dropping the units from interpretations: "adds 9.6" is meaningless without "thousand dollars per hundred square feet". Fourth, fitting a line to a curved scatterplot - least squares assumes the form is straight, and the residual topic ahead shows how to catch the failure. Fifth, interpreting a meaningless intercept. Sixth, predicting \(x\) from \(y\) with the \(y\)-on-\(x\) line: the response variable is a choice, and the choice has consequences.

Key takeaways

The least-squares line \(\hat{y} = b_0 + b_1x\) minimises squared vertical misses, with \(b_1 = r\,s_y/s_x\) and \(b_0 = \bar{y} - b_1\bar{x}\), and it must pass through \((\bar{x}, \bar{y})\) - the free check. The houses: \(b_1 = 0.8 \times 60/5 = 9.6\), \(b_0 = 108\), line \(\hat{y} = 108 + 9.6x\), about $9,600 per extra 100 square feet, intercept meaningless, prediction 338.4 thousand at 2,400 sq ft, extrapolation to 4,000 sq ft unsound. Predictions shrink toward the mean in standardised terms (\(k\) SDs in \(x\) predicts \(rk\) in \(y\)); regression is not symmetric; and the slope is association, never proof of cause.

Flashcards

Flashcard

ŷ = b0 + b1x

The regression line: predicts y from x

The hat marks a prediction.

Test your understanding

Question 1 of 5

For the line ŷ = 40 + 2.5x, what does 2.5 mean?

Question 2 of 5

With r = 0.8, sy = 60 and sx = 5, the least-squares slope is:

Question 3 of 5

With b1 = 9.6, x̄ = 20 and ȳ = 300, the intercept is:

Question 4 of 5

With ŷ = 108 + 9.6x and sizes running 15 to 25, the prediction at x = 24 is:

Question 5 of 5

A house sits 2 SDs above the mean size (r = 0.8). Its predicted price is:

← Previous topicNext topic →