Linear Algebra › Section F · Orthogonality and least squares › Topic F.1

Inner product, length, and orthogonality

Length, distance and right angles are easy to picture in the plane, but data often lives in many more dimensions. One simple operation, the inner product, carries all three ideas into every space of vectors, and the rest of this section builds on it.

Why vectors need geometry

So far, vectors in \(\mathbb{R}^n\) have been things to add, scale and combine. Many practical questions also ask how far apart two vectors are, or which vector in a subspace is closest to a given one. A system \(A\mathbf{x} = \mathbf{b}\) built from noisy measurements usually has no exact solution, and the sensible fallback is the \(\mathbf{x}\) that brings \(A\mathbf{x}\) nearest to \(\mathbf{b}\). To make "nearest" precise, \(\mathbb{R}^n\) needs a notion of length, of distance and of perpendicularity. All three come from the inner product.

The inner product

For vectors \(\mathbf{u}\) and \(\mathbf{v}\) in \(\mathbb{R}^n\), treat each as an \(n \times 1\) matrix. Then \(\mathbf{u}^T\mathbf{v}\) is a \(1 \times 1\) matrix, which we write as a plain number:

\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^T\mathbf{v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n

This number is the inner product, or dot product. Four properties follow from the definition:\(\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}\); \((\mathbf{u} + \mathbf{v}) \cdot \mathbf{w} = \mathbf{u} \cdot \mathbf{w} + \mathbf{v} \cdot \mathbf{w}\); \((c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})\); and \(\mathbf{u} \cdot \mathbf{u} \ge 0\), with equality only when \(\mathbf{u} = \mathbf{0}\). The result is always a scalar, never a vector.

Length and unit vectors

Because \(\mathbf{v} \cdot \mathbf{v}\) is never negative, its square root exists. The length, or norm, of \(\mathbf{v}\) is

\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}} = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}

In \(\mathbb{R}^2\) this is Pythagoras's theorem for the arrow from the origin to \((v_1, v_2)\). Scaling a vector scales its length by the absolute value of the scalar: \(\|c\mathbf{v}\| = |c|\,\|\mathbf{v}\|\). A unit vector has length 1. Dividing any non-zero vector by its own length gives a unit vector pointing the same way, a step called normalising.

Worked example: normalising a vector in R4

Find the unit vector in the direction of \(\mathbf{v} = (2, -1, 2, 4)\). 1. Square and add the entries: \(\mathbf{v} \cdot \mathbf{v} = 4 + 1 + 4 + 16 = 25\). 2. Take the square root: \(\|\mathbf{v}\| = 5\). 3. Divide: \(\mathbf{u} = \frac{1}{5}\mathbf{v} = (\frac{2}{5}, -\frac{1}{5}, \frac{2}{5}, \frac{4}{5})\). 4. Check: \(\mathbf{u} \cdot \mathbf{u} = \frac{4 + 1 + 4 + 16}{25} = 1\), so \(\mathbf{u}\) is a unit vector. It is easier to check \(\|\mathbf{u}\|^2 = 1\) than \(\|\mathbf{u}\| = 1\), because no square root is needed. The vector \(-\mathbf{u}\) is the other unit vector on the same line, pointing the opposite way.

Distance between vectors

On the number line, the distance between \(a\) and \(b\) is \(|a - b|\). The same idea works in any dimension: the distance between \(\mathbf{u}\) and \(\mathbf{v}\) is the length of their difference,

\operatorname{dist}(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|

For \(\mathbf{u} = (4, 1, -2)\) and \(\mathbf{v} = (1, 3, 2)\), the difference is \((3, -2, -4)\), so the distance is \(\sqrt{9 + 4 + 16} = \sqrt{29} \approx 5.39\). The order does not matter, since \(\mathbf{v} - \mathbf{u}\) has the same entries with opposite signs and the same length.

Orthogonal vectors

When are two arrows perpendicular? In the plane, \(\mathbf{u}\) and \(\mathbf{v}\) are perpendicular exactly when \(\mathbf{u}\) is as far from \(\mathbf{v}\) as it is from \(-\mathbf{v}\). Expanding with the inner product rules gives

\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2 + 2\,\mathbf{u} \cdot \mathbf{v}

and the same with \(-2\,\mathbf{u} \cdot \mathbf{v}\) for \(\|\mathbf{u} - \mathbf{v}\|^2\). The two agree only when \(\mathbf{u} \cdot \mathbf{v} = 0\). So the definition for any \(\mathbb{R}^n\) is: \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal when \(\mathbf{u} \cdot \mathbf{v} = 0\). The zero vector is orthogonal to every vector.

The Pythagorean theorem in Rn

The expansion on the previous card gives Pythagoras's theorem in every dimension: \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal if and only if

\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2

For \(\mathbf{p} = (2, 1, -1)\) and \(\mathbf{q} = (1, -1, 1)\), the inner product is \(2 - 1 - 1 = 0\). Their squared lengths are 6 and 3, and \(\mathbf{p} + \mathbf{q} = (3, 0, 0)\) has squared length 9, which is \(6 + 3\), as the theorem promises. If the inner product had been positive, the sum would have been longer than Pythagoras predicts; if negative, shorter.

Angles, Cauchy-Schwarz and the triangle inequality

In \(\mathbb{R}^2\) and \(\mathbb{R}^3\), the law of cosines shows that \(\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta\), where \(\theta\) is the angle between the arrows. In higher dimensions this formula becomes the definition of the angle. It makes sense because of the Cauchy-Schwarz inequality, \(|\mathbf{u} \cdot \mathbf{v}| \le \|\mathbf{u}\|\,\|\mathbf{v}\|\), which keeps the cosine between -1 and 1. A consequence is the triangle inequality, \(\|\mathbf{u} + \mathbf{v}\| \le \|\mathbf{u}\| + \|\mathbf{v}\|\): one side of a triangle is never longer than the other two together. Both are proved in the later topic "Inner product spaces".

Worked example: the angle between two vectors

Find the angle between \(\mathbf{u} = (1, 2, 2)\) and \(\mathbf{w} = (3, 0, 4)\). 1. Inner product: \(\mathbf{u} \cdot \mathbf{w} = 3 + 0 + 8 = 11\). 2. Lengths: \(\|\mathbf{u}\| = \sqrt{9} = 3\) and \(\|\mathbf{w}\| = \sqrt{25} = 5\). 3. Cosine: \(\cos\theta = \frac{11}{3 \cdot 5} = \frac{11}{15} \approx 0.733\). 4. Angle: \(\theta = \arccos(0.733) \approx 42.8^\circ\). 5. Check: the inner product is positive, so the angle is acute, and \(11 \le 15\), as Cauchy-Schwarz requires. The triangle inequality also holds: \(\mathbf{u} + \mathbf{w} = (4, 2, 6)\) has length \(\sqrt{56} \approx 7.48\), less than \(3 + 5 = 8\).

Applied example: similarity of documents

Search engines and recommendation systems often turn documents into vectors of word counts. Suppose two short notes count the words "price", "delay", "refund" and "order" as \(\mathbf{x} = (3, 0, 1, 2)\) and \(\mathbf{y} = (1, 1, 0, 2)\). Their cosine similarity is \(\frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|\,\|\mathbf{y}\|} = \frac{7}{\sqrt{14}\sqrt{6}} \approx 0.76\), fairly similar. A note using only "delay", \((0, 4, 0, 0)\), has inner product 0 with \(\mathbf{x}\): orthogonal, no words in common. The same idea links to statistics: the correlation coefficient \(r\) is the cosine of the angle between the two data vectors after each has its mean subtracted.

Common mistakes to avoid

First, writing the inner product as a vector: \((1, 2) \cdot (3, 4)\) is the number 11, not \((3, 8)\). Second, forgetting the square root: \(\|\mathbf{v}\|\) is \(\sqrt{\mathbf{v} \cdot \mathbf{v}}\), so \((3, 4)\) has length 5, not 25. Third, normalising by the sum of the entries instead of the length. Fourth, assuming \(\|\mathbf{u} + \mathbf{v}\| = \|\mathbf{u}\| + \|\mathbf{v}\|\); equality holds only when one vector is a non-negative multiple of the other. Fifth, thinking a negative inner product means the vectors are orthogonal; it means the angle is obtuse. Only an inner product of exactly zero means orthogonal.

Key takeaways

The inner product \(\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^T\mathbf{v}\) multiplies matching entries and adds, giving a scalar. Length is \(\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}}\), with \(\|c\mathbf{v}\| = |c|\,\|\mathbf{v}\|\); dividing by the length gives a unit vector. Distance is \(\|\mathbf{u} - \mathbf{v}\|\). Vectors are orthogonal when their inner product is 0, which is exactly when Pythagoras's theorem holds for them. The angle comes from \(\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}\), kept valid by Cauchy-Schwarz, and lengths obey the triangle inequality.

Flashcards

Flashcard

u · v

uᵀv = u₁v₁ + u₂v₂ + ... + uₙvₙ, a scalar

Multiply matching entries, then add.

Test your understanding

Question 1 of 5

What is the inner product of (3, -2, 1) and (2, 4, 5)?

Question 2 of 5

What is the length of the vector (1, -2, 2, 4)?

Question 3 of 5

Which is the unit vector in the same direction as (6, 0, -8)?

Question 4 of 5

Vectors u and v in R⁵ satisfy ‖u + v‖² = ‖u‖² + ‖v‖². What must be true?

Question 5 of 5

What is the angle between u = (1, 1, 0) and v = (1, 0, 1)?

← Previous topicNext topic →