Linear Algebra › Section F · Orthogonality and least squares › Topic F.6

Least squares for linear models

The regression line of statistics is a least-squares problem in disguise. Writing it with a design matrix shows exactly where its formulas come from, and the same set-up fits curves, several predictors and any model whose unknown coefficients appear linearly.

Regression as a matrix problem

In statistics, the least-squares regression line \(\hat{y} = b_0 + b_1x\) is the line that makes the sum of squared residuals as small as possible, where each residual is observed minus predicted, \(y - \hat{y}\). Linear algebra writes the same problem as an inconsistent system \(A\mathbf{x} = \mathbf{b}\) and solves it with the normal equations from "Least-squares problems". Linear algebra texts usually call the coefficients \(\beta_0\) and \(\beta_1\), and the matrix \(X\) rather than \(A\). The payoff of the matrix view is that one method covers straight lines, curves and models with several explanatory variables.

Design matrix, parameters and observations

Suppose the data are \((x_1, y_1), \dots, (x_n, y_n)\) and the model is \(y = \beta_0 + \beta_1x\). If every point lay on the line, we would have \(\beta_0 + \beta_1x_i = y_i\) for each \(i\), that is, \(X\boldsymbol{\beta} = \mathbf{y}\) with

X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}, \quad \boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}

Here \(X\) is the design matrix, \(\boldsymbol{\beta}\) the parameter vector and \(\mathbf{y}\) the observation vector. Real data never lie exactly on a line, so we find the least-squares \(\hat{\boldsymbol{\beta}}\) and call \(\mathbf{e} = \mathbf{y} - X\hat{\boldsymbol{\beta}}\) the residual vector.

The normal equations for a line

The least-squares coefficients solve \(X^TX\hat{\boldsymbol{\beta}} = X^T\mathbf{y}\). Multiplying out shows what the entries are:

X^TX = \begin{bmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2 \end{bmatrix}, \qquad X^T\mathbf{y} = \begin{bmatrix} \sum y_i \\ \sum x_iy_i \end{bmatrix}

So only five sums are needed. As long as the \(x_i\) are not all equal, the two columns of \(X\) are independent and the solution is unique. Its slope agrees with the statistics formula \(b_1 = r\,\frac{s_y}{s_x}\), and its intercept with \(b_0 = \bar{y} - b_1\bar{x}\): they are two routes to the same line.

Worked example: fitting a line

Fit \(y = \beta_0 + \beta_1x\) to \((0, 1), (1, 2), (2, 4), (3, 4), (4, 7)\). 1. Sums: \(n = 5\), \(\sum x_i = 10\), \(\sum x_i^2 = 30\), \(\sum y_i = 18\), \(\sum x_iy_i = 0 + 2 + 8 + 12 + 28 = 50\). 2. Normal equations: \(5\beta_0 + 10\beta_1 = 18\) and \(10\beta_0 + 30\beta_1 = 50\). 3. Doubling the first and subtracting gives \(10\beta_1 = 14\), so \(\beta_1 = 1.4\), and then \(\beta_0 = \frac{18 - 14}{5} = 0.8\). 4. The line is \(\hat{y} = 0.8 + 1.4x\). 5. Residuals: \(0.2, -0.2, 0.4, -1, 0.6\). They add to 0, and the line passes through the point of means \((2, 3.6)\), since \(0.8 + 2.8 = 3.6\).

Why the residuals add to zero

The residual vector \(\mathbf{e}\) is orthogonal to every column of \(X\). One of those columns is all 1s, and the inner product of \(\mathbf{e}\) with it is simply the sum of the residuals, so that sum is 0. Dividing the first normal equation by \(n\) gives \(\hat{\beta}_0 + \hat{\beta}_1\bar{x} = \bar{y}\), which is why the line always passes through \((\bar{x}, \bar{y})\). Both facts, familiar from statistics, are consequences of orthogonality, and both depend on the model including an intercept. The size of \(\mathbf{e}\) also measures fit: in the example, \(\|\mathbf{e}\|^2 = 1.6\), and \(r^2 = 1 - \frac{1.6}{21.2} \approx 0.92\). Because squaring magnifies large misses, one outlying point can pull the whole fit, so plot the residuals too.

The mean-deviation form

Shift the explanatory variable so its mean is zero: \(x_i^* = x_i - \bar{x}\). The column of \(x^*\) values then adds to zero, so it is orthogonal to the column of 1s, and \(X^TX\) becomes diagonal. The normal equations separate: the intercept of the shifted model is \(\bar{y}\), and the slope is

\hat{\beta}_1 = \frac{\sum x_i^*y_i}{\sum (x_i^*)^2}

In the example, \(\bar{x} = 2\), so \(x^* = -2, -1, 0, 1, 2\). Then \(\sum x_i^*y_i = -2 - 2 + 0 + 4 + 14 = 14\) and \(\sum (x_i^*)^2 = 10\), giving slope 1.4 again, and \(\hat{y} = 3.6 + 1.4(x - 2)\), the same line. With orthogonal columns, each coefficient is simply the projection weight of \(\mathbf{y}\) on its own column, as in "Orthogonal projections".

The general linear model

Every example in this topic has the form

\mathbf{y} = X\boldsymbol{\beta} + \boldsymbol{\epsilon}

where \(\boldsymbol{\epsilon}\) holds the unknown errors, and the least-squares \(\hat{\boldsymbol{\beta}}\) always solves \(X^TX\hat{\boldsymbol{\beta}} = X^T\mathbf{y}\). The model is called linear because it is linear in the parameters, not because its graph is straight. Each column of \(X\) is any fixed function of the data: \(1\), \(x\), \(x^2\), \(\sin x\), or a second variable. A cost curve \(y = \beta_0 + \beta_1x + \beta_2x^2\) and a seasonal model \(y = \beta_0 + \beta_1\cos t + \beta_2\sin t\) both qualify. By contrast, \(y = \beta_0e^{\beta_1x}\) does not, because \(\beta_1\) sits inside the exponential.

Worked example: fitting a parabola

Fit \(y = \beta_0 + \beta_1x + \beta_2x^2\) to \((-2, 6), (-1, 2), (0, 1), (1, 2), (2, 5)\). 1. The rows of \(X\) are \((1, x_i, x_i^2)\), for example \((1, -2, 4)\). 2. Normal equations:

\begin{bmatrix} 5 & 0 & 10 \\ 0 & 10 & 0 \\ 10 & 0 & 34 \end{bmatrix}\hat{\boldsymbol{\beta}} = \begin{bmatrix} 16 \\ -2 \\ 48 \end{bmatrix}

3. The middle equation gives \(\beta_1 = -0.2\). Doubling the first gives \(10\beta_0 + 20\beta_2 = 32\); subtracting from the third leaves \(14\beta_2 = 16\), so \(\beta_2 = \frac{8}{7}\), and \(\beta_0 = \frac{16 - 80/7}{5} = \frac{32}{35}\). 4. So \(\hat{y} \approx 0.914 - 0.2x + 1.143x^2\). 5. Check at \(x = 0\): predicted 0.914, observed 1, residual \(0.086 = \frac{3}{35}\), small as expected.

Several explanatory variables

Multiple regression predicts \(y\) from two or more variables, say \(y = \beta_0 + \beta_1u + \beta_2v\) for the price of a house from its floor area \(u\) and age \(v\). The design matrix simply gains a column per variable: row \(i\) is \((1, u_i, v_i)\). Nothing else changes. The normal equations are now \(3 \times 3\), the residuals still add to zero, and the solution is unique as long as no column of \(X\) is a combination of the others. When two variables are nearly proportional, the columns are nearly dependent, \(X^TX\) is close to singular, and the estimated coefficients become unstable: small changes in the data swing them widely. The QR route keeps the arithmetic accurate, but no method removes that sensitivity.

Common mistakes to avoid

First, leaving out the column of 1s; that forces the fit through the origin, and the residuals no longer need to add to zero. Second, swapping the order of columns and parameters, so that \(\beta_0\) multiplies the \(x\) column. Third, putting \(y\) values into \(X\); the observations belong only in \(\mathbf{y}\). Fourth, thinking a curved model needs a different method; a parabola is still linear in its parameters. Fifth, measuring residuals perpendicular to the line; least squares uses vertical misses, observed minus predicted, which is why regressing \(x\) on \(y\) gives a different line.

Key takeaways

A linear model is \(\mathbf{y} = X\boldsymbol{\beta} + \boldsymbol{\epsilon}\), with a design matrix \(X\), parameter vector \(\boldsymbol{\beta}\) and observation vector \(\mathbf{y}\). The least-squares coefficients solve \(X^TX\hat{\boldsymbol{\beta}} = X^T\mathbf{y}\), and the residual vector \(\mathbf{e} = \mathbf{y} - X\hat{\boldsymbol{\beta}}\) is orthogonal to every column of \(X\). With a column of 1s, the residuals add to zero and the fit passes through the means. Centring \(x\) makes the columns orthogonal. Polynomials, trigonometric terms and several variables all fit the same way, because the model only has to be linear in its parameters.

Flashcards

Flashcard

Design matrix X

Row i holds the model's terms at data point i, such as (1, xᵢ)

The column of 1s carries the intercept.

Test your understanding

Question 1 of 5

To fit y = β₀ + β₁x to the points (1, 4), (2, 5), (4, 9), which design matrix X is used?

Question 2 of 5

What is the least-squares line for the points (0, 1), (1, 3), (2, 2), (3, 6)?

Question 3 of 5

A model with an intercept is fitted by least squares. Which statement about its residuals always holds?

Question 4 of 5

Why does replacing each xᵢ by xᵢ - x̄ simplify fitting a line?

Question 5 of 5

Which model can be fitted directly by linear least squares in the form y = Xβ + ε?

← Previous topicNext topic →