Linear Algebra › Section F · Orthogonality and least squares › Topic F.1
Inner product, length, and orthogonality
Length, distance and right angles are easy to picture in the plane, but data often lives in many more dimensions. One simple operation, the inner product, carries all three ideas into every space of vectors, and the rest of this section builds on it.
Why vectors need geometry
So far, vectors in \(\mathbb{R}^n\) have been things to add, scale and combine. Many practical questions also ask how far apart two vectors are, or which vector in a subspace is closest to a given one. A system \(A\mathbf{x} = \mathbf{b}\) built from noisy measurements usually has no exact solution, and the sensible fallback is the \(\mathbf{x}\) that brings \(A\mathbf{x}\) nearest to \(\mathbf{b}\). To make "nearest" precise, \(\mathbb{R}^n\) needs a notion of length, of distance and of perpendicularity. All three come from the inner product.
The inner product
For vectors \(\mathbf{u}\) and \(\mathbf{v}\) in \(\mathbb{R}^n\), treat each as an \(n \times 1\) matrix. Then \(\mathbf{u}^T\mathbf{v}\) is a \(1 \times 1\) matrix, which we write as a plain number:
This number is the inner product, or dot product. Four properties follow from the definition:\(\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}\); \((\mathbf{u} + \mathbf{v}) \cdot \mathbf{w} = \mathbf{u} \cdot \mathbf{w} + \mathbf{v} \cdot \mathbf{w}\); \((c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})\); and \(\mathbf{u} \cdot \mathbf{u} \ge 0\), with equality only when \(\mathbf{u} = \mathbf{0}\). The result is always a scalar, never a vector.
Length and unit vectors
Because \(\mathbf{v} \cdot \mathbf{v}\) is never negative, its square root exists. The length, or norm, of \(\mathbf{v}\) is
In \(\mathbb{R}^2\) this is Pythagoras's theorem for the arrow from the origin to \((v_1, v_2)\). Scaling a vector scales its length by the absolute value of the scalar: \(\|c\mathbf{v}\| = |c|\,\|\mathbf{v}\|\). A unit vector has length 1. Dividing any non-zero vector by its own length gives a unit vector pointing the same way, a step called normalising.
Worked example: normalising a vector in R4
Find the unit vector in the direction of \(\mathbf{v} = (2, -1, 2, 4)\). 1. Square and add the entries: \(\mathbf{v} \cdot \mathbf{v} = 4 + 1 + 4 + 16 = 25\). 2. Take the square root: \(\|\mathbf{v}\| = 5\). 3. Divide: \(\mathbf{u} = \frac{1}{5}\mathbf{v} = (\frac{2}{5}, -\frac{1}{5}, \frac{2}{5}, \frac{4}{5})\). 4. Check: \(\mathbf{u} \cdot \mathbf{u} = \frac{4 + 1 + 4 + 16}{25} = 1\), so \(\mathbf{u}\) is a unit vector. It is easier to check \(\|\mathbf{u}\|^2 = 1\) than \(\|\mathbf{u}\| = 1\), because no square root is needed. The vector \(-\mathbf{u}\) is the other unit vector on the same line, pointing the opposite way.
Distance between vectors
On the number line, the distance between \(a\) and \(b\) is \(|a - b|\). The same idea works in any dimension: the distance between \(\mathbf{u}\) and \(\mathbf{v}\) is the length of their difference,
For \(\mathbf{u} = (4, 1, -2)\) and \(\mathbf{v} = (1, 3, 2)\), the difference is \((3, -2, -4)\), so the distance is \(\sqrt{9 + 4 + 16} = \sqrt{29} \approx 5.39\). The order does not matter, since \(\mathbf{v} - \mathbf{u}\) has the same entries with opposite signs and the same length.
Orthogonal vectors
When are two arrows perpendicular? In the plane, \(\mathbf{u}\) and \(\mathbf{v}\) are perpendicular exactly when \(\mathbf{u}\) is as far from \(\mathbf{v}\) as it is from \(-\mathbf{v}\). Expanding with the inner product rules gives
and the same with \(-2\,\mathbf{u} \cdot \mathbf{v}\) for \(\|\mathbf{u} - \mathbf{v}\|^2\). The two agree only when \(\mathbf{u} \cdot \mathbf{v} = 0\). So the definition for any \(\mathbb{R}^n\) is: \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal when \(\mathbf{u} \cdot \mathbf{v} = 0\). The zero vector is orthogonal to every vector.
The Pythagorean theorem in Rn
The expansion on the previous card gives Pythagoras's theorem in every dimension: \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal if and only if
For \(\mathbf{p} = (2, 1, -1)\) and \(\mathbf{q} = (1, -1, 1)\), the inner product is \(2 - 1 - 1 = 0\). Their squared lengths are 6 and 3, and \(\mathbf{p} + \mathbf{q} = (3, 0, 0)\) has squared length 9, which is \(6 + 3\), as the theorem promises. If the inner product had been positive, the sum would have been longer than Pythagoras predicts; if negative, shorter.
Angles, Cauchy-Schwarz and the triangle inequality
In \(\mathbb{R}^2\) and \(\mathbb{R}^3\), the law of cosines shows that \(\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta\), where \(\theta\) is the angle between the arrows. In higher dimensions this formula becomes the definition of the angle. It makes sense because of the Cauchy-Schwarz inequality, \(|\mathbf{u} \cdot \mathbf{v}| \le \|\mathbf{u}\|\,\|\mathbf{v}\|\), which keeps the cosine between -1 and 1. A consequence is the triangle inequality, \(\|\mathbf{u} + \mathbf{v}\| \le \|\mathbf{u}\| + \|\mathbf{v}\|\): one side of a triangle is never longer than the other two together. Both are proved in the later topic "Inner product spaces".
Worked example: the angle between two vectors
Find the angle between \(\mathbf{u} = (1, 2, 2)\) and \(\mathbf{w} = (3, 0, 4)\). 1. Inner product: \(\mathbf{u} \cdot \mathbf{w} = 3 + 0 + 8 = 11\). 2. Lengths: \(\|\mathbf{u}\| = \sqrt{9} = 3\) and \(\|\mathbf{w}\| = \sqrt{25} = 5\). 3. Cosine: \(\cos\theta = \frac{11}{3 \cdot 5} = \frac{11}{15} \approx 0.733\). 4. Angle: \(\theta = \arccos(0.733) \approx 42.8^\circ\). 5. Check: the inner product is positive, so the angle is acute, and \(11 \le 15\), as Cauchy-Schwarz requires. The triangle inequality also holds: \(\mathbf{u} + \mathbf{w} = (4, 2, 6)\) has length \(\sqrt{56} \approx 7.48\), less than \(3 + 5 = 8\).
Applied example: similarity of documents
Search engines and recommendation systems often turn documents into vectors of word counts. Suppose two short notes count the words "price", "delay", "refund" and "order" as \(\mathbf{x} = (3, 0, 1, 2)\) and \(\mathbf{y} = (1, 1, 0, 2)\). Their cosine similarity is \(\frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|\,\|\mathbf{y}\|} = \frac{7}{\sqrt{14}\sqrt{6}} \approx 0.76\), fairly similar. A note using only "delay", \((0, 4, 0, 0)\), has inner product 0 with \(\mathbf{x}\): orthogonal, no words in common. The same idea links to statistics: the correlation coefficient \(r\) is the cosine of the angle between the two data vectors after each has its mean subtracted.
Common mistakes to avoid
First, writing the inner product as a vector: \((1, 2) \cdot (3, 4)\) is the number 11, not \((3, 8)\). Second, forgetting the square root: \(\|\mathbf{v}\|\) is \(\sqrt{\mathbf{v} \cdot \mathbf{v}}\), so \((3, 4)\) has length 5, not 25. Third, normalising by the sum of the entries instead of the length. Fourth, assuming \(\|\mathbf{u} + \mathbf{v}\| = \|\mathbf{u}\| + \|\mathbf{v}\|\); equality holds only when one vector is a non-negative multiple of the other. Fifth, thinking a negative inner product means the vectors are orthogonal; it means the angle is obtuse. Only an inner product of exactly zero means orthogonal.
Key takeaways
The inner product \(\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^T\mathbf{v}\) multiplies matching entries and adds, giving a scalar. Length is \(\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}}\), with \(\|c\mathbf{v}\| = |c|\,\|\mathbf{v}\|\); dividing by the length gives a unit vector. Distance is \(\|\mathbf{u} - \mathbf{v}\|\). Vectors are orthogonal when their inner product is 0, which is exactly when Pythagoras's theorem holds for them. The angle comes from \(\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}\), kept valid by Cauchy-Schwarz, and lengths obey the triangle inequality.
Flashcards
Flashcard
uᵀv = u₁v₁ + u₂v₂ + ... + uₙvₙ, a scalar
Test your understanding
Question 1 of 5
What is the inner product of (3, -2, 1) and (2, 4, 5)?
Question 2 of 5
What is the length of the vector (1, -2, 2, 4)?
Question 3 of 5
Which is the unit vector in the same direction as (6, 0, -8)?
Question 4 of 5
Vectors u and v in R⁵ satisfy ‖u + v‖² = ‖u‖² + ‖v‖². What must be true?
Question 5 of 5
What is the angle between u = (1, 1, 0) and v = (1, 0, 1)?
