Linear Algebra › Section F · Orthogonality and least squares › Topic F.2
Orthogonal sets and complements
When the vectors of a basis meet at right angles, finding coordinates stops needing row reduction: each weight comes from one quick division. This topic shows why, meets the matrices whose columns behave this way, and describes everything perpendicular to a subspace.
Why right angles save work
Writing a vector in terms of a basis normally means solving a linear system. With the standard basis of \(\mathbb{R}^n\) there is nothing to solve: the coordinates are just the entries. The standard basis is special because its vectors are mutually perpendicular. This topic shows that any basis with that property, not only the standard one, gives its coordinates almost for free, using the inner product from the previous topic. It also introduces the orthogonal complement, the set of all vectors perpendicular to a subspace.
Orthogonal sets are independent
A set \(\{\mathbf{u}_1, \dots, \mathbf{u}_p\}\) in \(\mathbb{R}^n\) is orthogonal when every pair of distinct vectors in it is orthogonal: \(\mathbf{u}_i \cdot \mathbf{u}_j = 0\) for \(i \ne j\). If none of the vectors is zero, the set is linearly independent. To see why, suppose \(c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p = \mathbf{0}\). Take the inner product of both sides with \(\mathbf{u}_1\). Every term except the first vanishes, leaving \(c_1(\mathbf{u}_1 \cdot \mathbf{u}_1) = 0\). Since \(\mathbf{u}_1 \ne \mathbf{0}\), this forces \(c_1 = 0\), and the same argument works for each weight. So \(n\) non-zero orthogonal vectors in \(\mathbb{R}^n\) always form a basis, called an orthogonal basis.
Coordinates in an orthogonal basis
Let \(\{\mathbf{u}_1, \dots, \mathbf{u}_p\}\) be an orthogonal basis of a subspace \(W\), and let \(\mathbf{y}\) be in \(W\), so \(\mathbf{y} = c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p\). Taking the inner product with \(\mathbf{u}_j\) kills every term except one: \(\mathbf{y} \cdot \mathbf{u}_j = c_j(\mathbf{u}_j \cdot \mathbf{u}_j)\). Therefore
Each weight is computed on its own, with no system to solve and no need to know the other weights. Remember the denominator: it is the squared length of \(\mathbf{u}_j\), and it equals 1 only for unit vectors.
Worked example: coordinates without row reduction
Take \(\mathbf{u}_1 = (1, 2, 1)\), \(\mathbf{u}_2 = (1, 0, -1)\), \(\mathbf{u}_3 = (1, -1, 1)\), and write \(\mathbf{y} = (5, 1, -1)\) in this basis. 1. Check orthogonality: \(\mathbf{u}_1 \cdot \mathbf{u}_2 = 1 + 0 - 1 = 0\), \(\mathbf{u}_1 \cdot \mathbf{u}_3 = 1 - 2 + 1 = 0\), \(\mathbf{u}_2 \cdot \mathbf{u}_3 = 1 + 0 - 1 = 0\). Three non-zero orthogonal vectors in \(\mathbb{R}^3\) form a basis. 2. Squared lengths: 6, 2 and 3. 3. Weights: \(c_1 = \frac{5 + 2 - 1}{6} = 1\), \(c_2 = \frac{5 + 0 + 1}{2} = 3\), \(c_3 = \frac{5 - 1 - 1}{3} = 1\). 4. Check: \(\mathbf{u}_1 + 3\mathbf{u}_2 + \mathbf{u}_3\) adds \((1, 2, 1)\), \((3, 0, -3)\) and \((1, -1, 1)\), giving \((5, 1, -1)\).
Orthonormal sets and the test UᵀU = I
An orthonormal set is an orthogonal set of unit vectors. Normalising each vector of an orthogonal basis gives an orthonormal basis, and then every denominator in the weight formula is 1, so \(c_j = \mathbf{y} \cdot \mathbf{u}_j\). There is a compact test. Put the vectors into the columns of a matrix \(U\). The \((i, j)\) entry of \(U^TU\) is \(\mathbf{u}_i \cdot \mathbf{u}_j\), so the columns are orthonormal exactly when
Such a \(U\) need not be square. Multiplying by it preserves geometry: \(\|U\mathbf{x}\| = \|\mathbf{x}\|\) and \((U\mathbf{x}) \cdot (U\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}\), because \((U\mathbf{x})^T(U\mathbf{y}) = \mathbf{x}^TU^TU\mathbf{y} = \mathbf{x}^T\mathbf{y}\).
Orthogonal matrices
A square matrix with orthonormal columns is called an orthogonal matrix, a name that is standard even though orthonormal would be more accurate. For such a \(Q\), \(Q^TQ = I\) means \(Q^{-1} = Q^T\), so inverting costs nothing, and the rows are orthonormal too. For example,
has columns of length 1 that are mutually orthogonal. It sends \(\mathbf{x} = (3, 0, -3)\) to \((0, 3, 3)\), and both have length \(3\sqrt{2}\). Orthogonal matrices describe rotations and reflections, have determinant 1 or -1, and are prized in computing because, preserving lengths, they do not magnify errors already present in the data.
Orthogonal complements
If \(W\) is a subspace of \(\mathbb{R}^n\), its orthogonal complement \(W^\perp\), read "W perp", is the set of all vectors orthogonal to every vector in \(W\). Four facts make it useful. A vector is in \(W^\perp\) if and only if it is orthogonal to every vector in a spanning set of \(W\), so only finitely many checks are needed. \(W^\perp\) is itself a subspace. The only vector in both \(W\) and \(W^\perp\) is \(\mathbf{0}\). And \(\dim W + \dim W^\perp = n\), with \((W^\perp)^\perp = W\). In \(\mathbb{R}^3\), the complement of a plane through the origin is its normal line, and the complement of that line is the plane again.
The row space, column space and null space
For an \(m \times n\) matrix \(A\), the equation \(A\mathbf{x} = \mathbf{0}\) says that the inner product of \(\mathbf{x}\) with every row of \(A\) is zero. So the null space is exactly the set of vectors orthogonal to the rows:
and likewise
The second statement is the first applied to \(A^T\), whose rows are the columns of \(A\). Together with the rank-nullity theorem from "Rank and the rank-nullity theorem", these give the dimension count above. The four subspaces \(\operatorname{Row} A\), \(\operatorname{Nul} A\), \(\operatorname{Col} A\) and \(\operatorname{Nul} A^T\) come in two perpendicular pairs, a picture Gilbert Strang popularised as the fundamental theorem of linear algebra.
Worked example: a basis for a complement
Let \(W\) be spanned by \((1, 1, 0, 2)\) and \((0, 1, 1, -1)\) in \(\mathbb{R}^4\). Find a basis for \(W^\perp\). 1. Make the spanning vectors the rows of \(A\), so \(W = \operatorname{Row} A\) and \(W^\perp = \operatorname{Nul} A\). 2. Row reduce:
3. With \(x_3\) and \(x_4\) free, \(x_1 = x_3 - 3x_4\) and \(x_2 = -x_3 + x_4\), giving basis vectors \((1, -1, 1, 0)\) and \((-3, 1, 0, 1)\). 4. Check: each has inner product 0 with both rows, for example \(-3 + 1 + 0 + 2 = 0\), and \(2 + 2 = 4\) dimensions, as expected.
Common mistakes to avoid
First, forgetting the denominator \(\mathbf{u}_j \cdot \mathbf{u}_j\) when the basis is orthogonal but not orthonormal. Second, checking only some pairs: a set is orthogonal only if every pair has inner product 0. Third, calling a square matrix with orthogonal but non-unit columns an orthogonal matrix; its inverse is not its transpose. Fourth, using \(Q^{-1} = Q^T\) for a non-square \(U\) with orthonormal columns: \(U^TU = I\) still holds, but \(UU^T\) is not \(I\). Fifth, mixing up the pairs: \(\operatorname{Nul} A\) is perpendicular to the row space, not the column space.
Key takeaways
An orthogonal set of non-zero vectors is linearly independent, so \(n\) of them form a basis of \(\mathbb{R}^n\). In an orthogonal basis, the weight on \(\mathbf{u}_j\) is \(\frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j}\), or just \(\mathbf{y} \cdot \mathbf{u}_j\) when the basis is orthonormal. Columns are orthonormal exactly when \(U^TU = I\), and then \(U\) preserves lengths and inner products. A square such matrix is orthogonal, with \(Q^{-1} = Q^T\). The complement \(W^\perp\) is a subspace with \(\dim W + \dim W^\perp = n\), and \((\operatorname{Row} A)^\perp = \operatorname{Nul} A\), \((\operatorname{Col} A)^\perp = \operatorname{Nul} A^T\).
Flashcards
Flashcard
Every pair of distinct vectors has inner product 0
Test your understanding
Question 1 of 5
{(1, 1), (1, -1)} is an orthogonal basis of R². When (5, 1) is written in this basis, what is the weight on (1, -1)?
Question 2 of 5
Three non-zero vectors in R³ are mutually orthogonal. Which statement must be true?
Question 3 of 5
Q is a 4 × 4 orthogonal matrix. What is Q⁻¹?
Question 4 of 5
W is a 3-dimensional subspace of R⁷. What is the dimension of W⊥?
Question 5 of 5
For an m × n matrix A, which subspace is the orthogonal complement of Col A?
