Linear Algebra › Section F · Orthogonality and least squares › Topic F.7

Inner product spaces

Length, angle and projection need only a few rules to work. Any operation that obeys those rules gives a vector space geometry, which lets us measure how far apart two functions or polynomials are and find the best approximation to one by others.

Geometry beyond Rn

Polynomials and continuous functions form vector spaces, as "Vector spaces and subspaces" showed, but so far they have had no lengths or angles. Yet questions such as "which straight line is closest to the curve \(t^2\) on an interval?" are natural. The way forward is to notice that every result about length, orthogonality and projection in \(\mathbb{R}^n\) used only four properties of the dot product. Any operation with those four properties, on any vector space, supports the same theory. Such an operation is called an inner product, and a vector space equipped with one is an inner product space.

The four axioms

An inner product on a vector space \(V\) assigns to each pair \(\mathbf{u}, \mathbf{v}\) a real number \(\langle \mathbf{u}, \mathbf{v} \rangle\) such that, for all \(\mathbf{u}, \mathbf{v}, \mathbf{w}\) in \(V\) and scalars \(c\): 1. \(\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle\). 2. \(\langle \mathbf{u} + \mathbf{v}, \mathbf{w} \rangle = \langle \mathbf{u}, \mathbf{w} \rangle + \langle \mathbf{v}, \mathbf{w} \rangle\). 3. \(\langle c\mathbf{u}, \mathbf{v} \rangle = c\langle \mathbf{u}, \mathbf{v} \rangle\). 4. \(\langle \mathbf{u}, \mathbf{u} \rangle \ge 0\), and \(\langle \mathbf{u}, \mathbf{u} \rangle = 0\) only when \(\mathbf{u} = \mathbf{0}\). Then define length \(\|\mathbf{v}\| = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle}\), distance \(\|\mathbf{u} - \mathbf{v}\|\), and orthogonality by \(\langle \mathbf{u}, \mathbf{v} \rangle = 0\), exactly as in \(\mathbb{R}^n\).

A weighted inner product on R2

The dot product is not the only inner product on \(\mathbb{R}^2\). For example,

\langle \mathbf{u}, \mathbf{v} \rangle = 2u_1v_1 + 3u_2v_2

satisfies all four axioms: the first three follow from ordinary algebra, and \(\langle \mathbf{u}, \mathbf{u} \rangle = 2u_1^2 + 3u_2^2\) is positive unless both entries are zero. The weights must be positive, or axiom 4 fails. Orthogonality now depends on the inner product chosen. The vectors \((3, -2)\) and \((1, 1)\) have dot product 1, so they are not perpendicular in the usual sense, but \(2(3)(1) + 3(-2)(1) = 0\), so they are orthogonal for this weighted inner product. Weights like these appear when some measurements are trusted more than others.

Polynomials: an inner product by evaluation

On \(\mathbb{P}_n\), the polynomials of degree at most \(n\), choose \(n + 1\) distinct numbers \(t_0, \dots, t_n\) and define

\langle p, q \rangle = \sum_{i=0}^{n} p(t_i)\,q(t_i)

This is the dot product of the vectors of values. Axiom 4 holds because a non-zero polynomial of degree at most \(n\) has at most \(n\) roots, so it cannot vanish at all \(n + 1\) points. In \(\mathbb{P}_2\) with points 0, 1, 2, the polynomial \(p(t) = 1 + t\) has values \((1, 2, 3)\) and \(q(t) = t^2\) has values \((0, 1, 4)\). So \(\langle p, q \rangle = 0 + 2 + 12 = 14\) and \(\|p\| = \sqrt{1 + 4 + 9} = \sqrt{14}\).

Worked example: Gram-Schmidt on polynomials

Use the evaluation inner product on \(\mathbb{P}_2\) at 0, 1, 2 to orthogonalise \(1, t, t^2\). Work with value vectors: \(1 \to (1, 1, 1)\), \(t \to (0, 1, 2)\), \(t^2 \to (0, 1, 4)\). 1. \(p_0 = 1\), with \(\|p_0\|^2 = 3\). 2. \(\langle t, 1 \rangle = 3\), so \(p_1 = t - \frac{3}{3} = t - 1\), with values \((-1, 0, 1)\) and \(\|p_1\|^2 = 2\). 3. \(\langle t^2, 1 \rangle = 5\) and \(\langle t^2, p_1 \rangle = 0 + 0 + 4 = 4\), so

p_2 = t^2 - \tfrac{5}{3} - \tfrac{4}{2}(t - 1) = t^2 - 2t + \tfrac{1}{3}

4. Check: \(p_2\) has values \((\frac{1}{3}, -\frac{2}{3}, \frac{1}{3})\), whose inner products with \((1, 1, 1)\) and \((-1, 0, 1)\) are both 0.

Functions: an inner product by integration

On \(C[a, b]\), the continuous functions on the interval \([a, b]\), define

\langle f, g \rangle = \int_a^b f(t)g(t)\,dt

The first three axioms follow from properties of integrals. For axiom 4, \(\int_a^b f(t)^2\,dt \ge 0\), and if a continuous \(f\) were non-zero somewhere, \(f^2\) would be positive on a small interval around that point, making the integral positive. This inner product turns the integral of a squared difference into a squared distance. Applying Gram-Schmidt to \(1, t, t^2\) with the interval \([-1, 1]\) gives \(1\), \(t\) and \(t^2 - \frac{1}{3}\), the start of the Legendre polynomials, which are used in numerical integration and physics.

Worked example: the best line for a curve

In \(C[0, 1]\) with the integral inner product, find the line closest to \(f(t) = t^2\). 1. Orthogonal basis for the lines: \(\langle t, 1 \rangle = \frac{1}{2}\) and \(\langle 1, 1 \rangle = 1\), so Gram-Schmidt gives \(1\) and \(t - \frac{1}{2}\), with \(\|t - \frac{1}{2}\|^2 = \frac{1}{12}\). 2. Weights: \(\langle t^2, 1 \rangle = \frac{1}{3}\), and \(\langle t^2, t - \frac{1}{2} \rangle = \frac{1}{4} - \frac{1}{6} = \frac{1}{12}\), giving weight 1. 3. Projection: \(\hat{f}(t) = \frac{1}{3} + (t - \frac{1}{2}) = t - \frac{1}{6}\). 4. Check: the error \(t^2 - t + \frac{1}{6}\) has integral 0 against both 1 and \(t\), and its squared length is \(\frac{1}{180}\), a distance of about 0.075.

Everything carries over

Since the proofs in \(\mathbb{R}^n\) used only the axioms, the main results hold in every inner product space. Orthogonal sets of non-zero vectors are independent. Gram-Schmidt turns a basis of a finite-dimensional subspace \(W\) into an orthogonal one. The projection \(\operatorname{proj}_W \mathbf{y}\), computed with the same formula using \(\langle \cdot, \cdot \rangle\) in place of the dot product, is the unique element of \(W\) closest to \(\mathbf{y}\): the best approximation theorem. The Pythagorean theorem holds too. The previous worked example was exactly this: \(t - \frac{1}{6}\) is the best approximation to \(t^2\) from the subspace of lines, measured by the integral distance.

Proving Cauchy-Schwarz and the triangle inequality

Cauchy-Schwarz states \(|\langle \mathbf{u}, \mathbf{v} \rangle| \le \|\mathbf{u}\|\,\|\mathbf{v}\|\). If \(\mathbf{u} = \mathbf{0}\), both sides are 0. Otherwise, project \(\mathbf{v}\) onto the line through \(\mathbf{u}\). The projection has length \(\frac{|\langle \mathbf{v}, \mathbf{u} \rangle|}{\|\mathbf{u}\|}\), and by Pythagoras a projection is never longer than the vector itself, so this is at most \(\|\mathbf{v}\|\). Multiplying by \(\|\mathbf{u}\|\) finishes the proof. For the triangle inequality, expand and then apply Cauchy-Schwarz to the middle term:

\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + 2\langle \mathbf{u}, \mathbf{v} \rangle + \|\mathbf{v}\|^2
\le \|\mathbf{u}\|^2 + 2\|\mathbf{u}\|\,\|\mathbf{v}\| + \|\mathbf{v}\|^2

which is \((\|\mathbf{u}\| + \|\mathbf{v}\|)^2\). Taking square roots gives \(\|\mathbf{u} + \mathbf{v}\| \le \|\mathbf{u}\| + \|\mathbf{v}\|\).

Common mistakes to avoid

First, using too few evaluation points: on \(\mathbb{P}_2\), evaluating only at 0 and 1 fails axiom 4, because \(t(t - 1)\) is non-zero yet has length 0. Second, allowing a zero or negative weight in a weighted inner product. Third, assuming that vectors orthogonal for one inner product are orthogonal for another. Fourth, reading \(\langle f, g \rangle\) as the function \(f(t)g(t)\); it is a single number, the integral of that product. Fifth, forgetting the square root: \(\|t\|\) in \(C[0, 1]\) is \(\frac{1}{\sqrt{3}}\), not \(\frac{1}{3}\).

Key takeaways

An inner product is any pairing \(\langle \mathbf{u}, \mathbf{v} \rangle\) that is symmetric, additive, compatible with scalars and positive for non-zero vectors. It defines length, distance and orthogonality. Examples are weighted dot products with positive weights, evaluation at \(n + 1\) distinct points on \(\mathbb{P}_n\), and \(\int_a^b f(t)g(t)\,dt\) on \(C[a, b]\). Orthogonality depends on the inner product chosen. Gram-Schmidt, projections and the best approximation theorem work unchanged, so \(t - \frac{1}{6}\) is the best line for \(t^2\) on \([0, 1]\). Cauchy-Schwarz and the triangle inequality hold in every inner product space.

Flashcards

Flashcard

Inner product space

A vector space with an inner product ⟨u, v⟩

Gives lengths, distances and angles.

Test your understanding

Question 1 of 5

R² has the inner product ⟨u, v⟩ = 2u₁v₁ + 3u₂v₂. Which vector is orthogonal to (1, 2) for this inner product?

Question 2 of 5

Which of these is NOT an inner product on P₂, the polynomials of degree at most 2?

Question 3 of 5

In C[0, 1] with ⟨f, g⟩ = ∫ f(t)g(t) dt from 0 to 1, what is the length of f(t) = t?

Question 4 of 5

In an inner product space, ‖u‖ = 2 and ‖v‖ = 3. Which value is impossible for ⟨u, v⟩?

Question 5 of 5

Using ⟨f, g⟩ = ∫ f(t)g(t) dt on [0, 1], which line is the best approximation to t²?

← Previous topicNext topic →