Linear Algebra › Section F · Orthogonality and least squares › Topic F.3

Orthogonal projections

Given a vector and a subspace, which vector in the subspace is closest? The answer is the orthogonal projection, the shadow the vector casts straight down onto the subspace, and it is the engine behind least squares, data fitting and signal approximation.

The closest point in a subspace

Stand at a point above a flat floor and drop a plumb line: it meets the floor at the point nearest to you, and the line itself is perpendicular to the floor. Linear algebra makes the same idea work in \(\mathbb{R}^n\). Given a vector \(\mathbf{y}\) and a subspace \(W\), we split \(\mathbf{y}\) into a piece inside \(W\) and a piece perpendicular to \(W\). The piece inside is the orthogonal projection, and it turns out to be the vector of \(W\) closest to \(\mathbf{y}\). The previous topic, "Orthogonal sets and complements", supplies the tools: orthogonal bases and \(W^\perp\).

Projection onto a line

Let \(L\) be the line through the origin spanned by a non-zero vector \(\mathbf{u}\). We want \(\hat{\mathbf{y}} = c\mathbf{u}\) with \(\mathbf{y} - \hat{\mathbf{y}}\) orthogonal to \(\mathbf{u}\). Setting \((\mathbf{y} - c\mathbf{u}) \cdot \mathbf{u} = 0\) gives \(c = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\), so

\operatorname{proj}_L \mathbf{y} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\,\mathbf{u}

Replacing \(\mathbf{u}\) by any non-zero multiple gives the same answer, so the projection depends only on the line. For \(\mathbf{y} = (7, 1)\) and \(\mathbf{u} = (1, 1)\), \(c = \frac{8}{2} = 4\), so \(\hat{\mathbf{y}} = (4, 4)\) and \(\mathbf{y} - \hat{\mathbf{y}} = (3, -3)\), which is orthogonal to \(\mathbf{u}\).

The orthogonal decomposition theorem

Now let \(W\) be any subspace of \(\mathbb{R}^n\). Every \(\mathbf{y}\) in \(\mathbb{R}^n\) can be written in exactly one way as \(\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}\), with \(\hat{\mathbf{y}}\) in \(W\) and \(\mathbf{z}\) in \(W^\perp\). If \(\{\mathbf{u}_1, \dots, \mathbf{u}_p\}\) is any orthogonal basis of \(W\), then

\hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\mathbf{u}_1 + \cdots + \frac{\mathbf{y} \cdot \mathbf{u}_p}{\mathbf{u}_p \cdot \mathbf{u}_p}\mathbf{u}_p

and \(\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}\). The vector \(\hat{\mathbf{y}}\) is written \(\operatorname{proj}_W \mathbf{y}\). Because the decomposition is unique, a different orthogonal basis of \(W\) gives the same \(\hat{\mathbf{y}}\). If \(\mathbf{y}\) is already in \(W\), its projection is \(\mathbf{y}\) itself.

Why the basis must be orthogonal

The formula adds up the projections of \(\mathbf{y}\) onto each basis line separately. That only works when the lines are mutually perpendicular. Check what happens to \(\mathbf{z}\): its inner product with \(\mathbf{u}_1\) is \(\mathbf{y} \cdot \mathbf{u}_1\) minus the first term's contribution, which cancels it exactly, minus the other terms' contributions, which are zero only because \(\mathbf{u}_j \cdot \mathbf{u}_1 = 0\). With a basis that is not orthogonal, those cross terms survive, \(\mathbf{z}\) is not perpendicular to \(W\), and the sum is not the projection. If you have a basis that is not orthogonal, first make it orthogonal; the next topic shows how.

Worked example: projecting onto a plane

Let \(W\) be spanned by \(\mathbf{u}_1 = (1, 1, 1)\) and \(\mathbf{u}_2 = (1, -2, 1)\), and let \(\mathbf{y} = (3, 1, 5)\). 1. Check the basis: \(\mathbf{u}_1 \cdot \mathbf{u}_2 = 1 - 2 + 1 = 0\), so the formula applies. 2. Weights: \(\frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1} = \frac{9}{3} = 3\) and \(\frac{\mathbf{y} \cdot \mathbf{u}_2}{\mathbf{u}_2 \cdot \mathbf{u}_2} = \frac{6}{6} = 1\). 3. Projection: \(\hat{\mathbf{y}} = 3(1, 1, 1) + (1, -2, 1) = (4, 1, 4)\). 4. Perpendicular part: \(\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = (-1, 0, 1)\). 5. Check: \(\mathbf{z} \cdot \mathbf{u}_1 = -1 + 0 + 1 = 0\) and \(\mathbf{z} \cdot \mathbf{u}_2 = -1 + 0 + 1 = 0\), so \(\mathbf{z}\) is in \(W^\perp\), as the theorem requires.

The best approximation theorem

The projection \(\hat{\mathbf{y}}\) is the closest point of \(W\) to \(\mathbf{y}\): for every \(\mathbf{v}\) in \(W\) other than \(\hat{\mathbf{y}}\),

\|\mathbf{y} - \hat{\mathbf{y}}\| < \|\mathbf{y} - \mathbf{v}\|

The proof is Pythagoras. Write \(\mathbf{y} - \mathbf{v} = (\mathbf{y} - \hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{v})\). The first piece is in \(W^\perp\) and the second is in \(W\), so they are orthogonal, and \(\|\mathbf{y} - \mathbf{v}\|^2 = \|\mathbf{y} - \hat{\mathbf{y}}\|^2 + \|\hat{\mathbf{y}} - \mathbf{v}\|^2\). The last term is positive when \(\mathbf{v} \ne \hat{\mathbf{y}}\). For this reason \(\hat{\mathbf{y}}\) is also called the best approximation to \(\mathbf{y}\) by elements of \(W\), and \(\|\mathbf{y} - \hat{\mathbf{y}}\|\) is the distance from \(\mathbf{y}\) to \(W\).

Worked example: distance to a plane

Continue with \(\mathbf{y} = (3, 1, 5)\) and the plane \(W\) above. 1. The distance from \(\mathbf{y}\) to \(W\) is \(\|\mathbf{z}\| = \|(-1, 0, 1)\| = \sqrt{2} \approx 1.41\). 2. Compare another point of \(W\), say \(\mathbf{v} = 2\mathbf{u}_1 = (2, 2, 2)\). Then \(\mathbf{y} - \mathbf{v} = (1, -1, 3)\), with squared length 11. 3. The theorem predicts \(11 = \|\mathbf{z}\|^2 + \|\hat{\mathbf{y}} - \mathbf{v}\|^2\). Here \(\hat{\mathbf{y}} - \mathbf{v} = (2, -1, 2)\), with squared length 9, and \(2 + 9 = 11\). 4. So \(\mathbf{v}\) is \(\sqrt{11} \approx 3.32\) away, farther than \(\hat{\mathbf{y}}\), as it must be.

Projection matrices from orthonormal bases

If the basis of \(W\) is orthonormal, every denominator is 1 and \(\hat{\mathbf{y}} = (\mathbf{y} \cdot \mathbf{u}_1)\mathbf{u}_1 + \cdots + (\mathbf{y} \cdot \mathbf{u}_p)\mathbf{u}_p\). Put the \(\mathbf{u}_j\) in the columns of an \(n \times p\) matrix \(U\). Then \(U^T\mathbf{y}\) lists the weights and \(U\) recombines them, so

\operatorname{proj}_W \mathbf{y} = UU^T\mathbf{y}

The \(n \times n\) matrix \(P = UU^T\) is the projection matrix onto \(W\). It is symmetric, \(P^T = P\), and projecting twice changes nothing, so \(P^2 = P\). Contrast \(U^TU = I_p\), the small identity matrix; \(UU^T\) equals \(I_n\) only when \(W\) is all of \(\mathbb{R}^n\).

Worked example: a projection matrix

Let \(W\) have orthonormal basis \(\frac{1}{\sqrt{2}}(1, 0, 1)\) and \((0, 1, 0)\). 1. Multiply out \(UU^T\):

P = \begin{bmatrix} \frac{1}{2} & 0 & \frac{1}{2} \\ 0 & 1 & 0 \\ \frac{1}{2} & 0 & \frac{1}{2} \end{bmatrix}

2. Project \(\mathbf{y} = (2, 3, 4)\): \(P\mathbf{y} = (3, 3, 3)\). 3. Perpendicular part: \(\mathbf{y} - P\mathbf{y} = (-1, 0, 1)\), which has inner product 0 with both basis vectors. 4. Check \(P^2 = P\): the first row of \(P^2\) is \((\frac{1}{4} + \frac{1}{4}, 0, \frac{1}{4} + \frac{1}{4}) = (\frac{1}{2}, 0, \frac{1}{2})\), the same as the first row of \(P\).

Common mistakes to avoid

First, using the sum-of-projections formula with a basis that is not orthogonal; the result is not the projection. Second, dropping the denominators \(\mathbf{u}_j \cdot \mathbf{u}_j\) when the basis is orthogonal but not orthonormal. Third, reporting \(\|\hat{\mathbf{y}}\|\) as the distance to \(W\); the distance is \(\|\mathbf{y} - \hat{\mathbf{y}}\|\). Fourth, writing \(\|\mathbf{y} - \mathbf{v}\| = \|\mathbf{y} - \hat{\mathbf{y}}\| + \|\hat{\mathbf{y}} - \mathbf{v}\|\); Pythagoras adds squared lengths. Fifth, confusing \(UU^T\), the projection matrix, with \(U^TU\), which is the identity.

Key takeaways

Projection onto the line through \(\mathbf{u}\) is \(\frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\mathbf{u}\). Every \(\mathbf{y}\) splits uniquely as \(\hat{\mathbf{y}} + \mathbf{z}\), with \(\hat{\mathbf{y}}\) in \(W\) and \(\mathbf{z}\) in \(W^\perp\); with an orthogonal basis, \(\hat{\mathbf{y}}\) is the sum of the projections onto the basis lines. The best approximation theorem says \(\hat{\mathbf{y}}\) is the closest point of \(W\), and the distance from \(\mathbf{y}\) to \(W\) is \(\|\mathbf{y} - \hat{\mathbf{y}}\|\). With orthonormal columns in \(U\), \(\operatorname{proj}_W \mathbf{y} = UU^T\mathbf{y}\), and \(P = UU^T\) satisfies \(P^T = P\) and \(P^2 = P\).

Flashcards

Flashcard

proj onto a line

((y · u) / (u · u)) u

Any non-zero multiple of u gives the same result.

Test your understanding

Question 1 of 5

What is the orthogonal projection of y = (1, 5) onto the line spanned by u = (2, 1)?

Question 2 of 5

y = ŷ + z is the orthogonal decomposition of y with respect to a subspace W. Which statement is true?

Question 3 of 5

The projection of y = (6, 2, 3) onto a subspace W is (6, 2, 0). What is the distance from y to W?

Question 4 of 5

U has orthonormal columns and P = UUᵀ. Which equation is always true?

Question 5 of 5

ŷ is the orthogonal projection of y onto W, and v is a different vector in W. Which statement is true?

← Previous topicNext topic →