Linear Algebra › Section F · Orthogonality and least squares › Topic F.8
Applications of inner product spaces
Choosing the right inner product turns practical problems into projections: fitting data whose points deserve different amounts of trust, tracking trends with polynomials that do not interfere with each other, and building a function out of sine and cosine waves.
One idea, three applications
"Inner product spaces" showed that projections and the best approximation theorem work for any inner product. This topic puts that freedom to use. Each application below is a best-approximation problem: pick a vector space, pick an inner product that measures error the way the problem needs, and project. Weighted least squares changes the inner product on \(\mathbb{R}^n\). Trend analysis uses an evaluation inner product on polynomials. Fourier approximation uses the integral inner product on continuous functions. In every case the answer is the projection onto a subspace, and orthogonal bases make it easy to compute.
Weighted least squares
Ordinary least squares treats every data point equally. Often some measurements are more reliable than others, perhaps taken with a better instrument or averaged over more trials. Give point \(i\) a positive weight \(w_i\) and minimise the weighted sum of squared residuals \(\sum w_i(y_i - \hat{y}_i)^2\). This is the squared length of the residual vector for the weighted inner product \(\langle \mathbf{u}, \mathbf{v} \rangle = \sum w_iu_iv_i\). With \(W\) the diagonal matrix of weights, the projection argument gives the weighted normal equations
When all weights are equal, \(W\) cancels and ordinary least squares returns.
Worked example: trusting one point more
Fit a line to \((0, 0), (1, 3), (2, 2)\), where the middle reading is twice as reliable, so the weights are 1, 2, 1. 1. With rows \((1, x_i)\) in \(X\):
and \(X^TW\mathbf{y} = (8, 10)\), from \(0 + 6 + 2\) and \(0 + 6 + 4\). 2. Solve \(4\beta_0 + 4\beta_1 = 8\) and \(4\beta_0 + 6\beta_1 = 10\): \(\beta_1 = 1\) and \(\beta_0 = 1\), so \(\hat{y} = 1 + x\). 3. Compare ordinary least squares: \(3\beta_0 + 3\beta_1 = 5\) and \(3\beta_0 + 5\beta_1 = 7\) give \(\hat{y} = \frac{2}{3} + x\). 4. Check: at \(x = 1\) the weighted line predicts 2, nearer the trusted reading of 3 than the ordinary line's \(\frac{5}{3}\), and the weighted residuals \(-1, 1, -1\) satisfy \(\sum w_ie_i = -1 + 2 - 1 = 0\).
Trend analysis with orthogonal polynomials
Suppose a quantity is recorded at equally spaced times, relabelled so they are centred at zero, such as \(t = -2, -1, 0, 1, 2\). To fit a polynomial trend, use the evaluation inner product \(\langle p, q \rangle = \sum p(t_i)q(t_i)\) and the orthogonal polynomials it produces. At these five points, \(p_0 = 1\), \(p_1 = t\) and \(p_2 = t^2 - 2\) are orthogonal: their value vectors \((1, 1, 1, 1, 1)\), \((-2, -1, 0, 1, 2)\) and \((2, -1, -2, -1, 2)\) have pairwise inner products 0. The best trend of degree 2 is then a sum of three independent projections, \(\hat{y} = c_0p_0 + c_1p_1 + c_2p_2\) with \(c_k = \frac{\langle \mathbf{y}, p_k \rangle}{\langle p_k, p_k \rangle}\). Raising the degree adds a new term and leaves the earlier coefficients unchanged.
Worked example: a quadratic trend
Readings at \(t = -2, -1, 0, 1, 2\) are \(\mathbf{y} = (6, 2, 1, 2, 5)\). 1. Squared lengths: \(\langle p_0, p_0 \rangle = 5\), \(\langle p_1, p_1 \rangle = 10\), \(\langle p_2, p_2 \rangle = 4 + 1 + 4 + 1 + 4 = 14\). 2. Coefficients: \(c_0 = \frac{16}{5}\); \(c_1 = \frac{-12 - 2 + 0 + 2 + 10}{10} = -0.2\); \(c_2 = \frac{12 - 2 - 2 - 2 + 10}{14} = \frac{8}{7}\). 3. The linear trend is \(3.2 - 0.2t\), nearly flat. 4. The quadratic trend is \(3.2 - 0.2t + \frac{8}{7}(t^2 - 2)\), which simplifies to \(\frac{32}{35} - 0.2t + \frac{8}{7}t^2\). 5. Check: this is exactly the parabola found with a full \(3 \times 3\) system in "Least squares for linear models", here obtained by three divisions.
Fourier approximation
Many signals, from sound waves to daily temperature cycles, repeat. On \(C[-\pi, \pi]\) with \(\langle f, g \rangle = \int_{-\pi}^{\pi} f(t)g(t)\,dt\), the functions
form an orthogonal set. For example, \(\int_{-\pi}^{\pi} \cos 2t \cos 3t\,dt = 0\), as product-to-sum identities show. Their squared lengths are \(\langle 1, 1 \rangle = 2\pi\) and \(\langle \cos kt, \cos kt \rangle = \langle \sin kt, \sin kt \rangle = \pi\) for \(k \ge 1\). The projection of \(f\) onto their span is the \(n\)th-order Fourier approximation of \(f\), the trigonometric polynomial of order \(n\) closest to \(f\) in this inner product.
The Fourier coefficients
Dividing each inner product by the right squared length gives the projection weights, the Fourier coefficients:
and the approximation is \(\frac{a_0}{2} + \sum_{k=1}^{n} (a_k\cos kt + b_k\sin kt)\). The constant term is written \(\frac{a_0}{2}\) because \(\langle 1, 1 \rangle = 2\pi\), not \(\pi\), and it equals the average value of \(f\) on the interval. An even function has every \(b_k = 0\), and an odd function has every \(a_k = 0\). Because the set is orthogonal, raising \(n\) adds new terms without changing the old ones.
Worked example: approximating a V shape
Approximate \(f(t) = |t|\) on \([-\pi, \pi]\). 1. \(f\) is even, so every \(b_k = 0\). 2. Average value: \(\frac{a_0}{2} = \frac{1}{2\pi}\int_{-\pi}^{\pi} |t|\,dt = \frac{\pi^2}{2\pi} = \frac{\pi}{2}\). 3. Integration by parts gives \(a_k = \frac{2}{\pi k^2}((-1)^k - 1)\): \(a_1 = -\frac{4}{\pi}\), \(a_2 = 0\), \(a_3 = -\frac{4}{9\pi}\). 4. Third-order approximation: \(\frac{\pi}{2} - \frac{4}{\pi}\cos t - \frac{4}{9\pi}\cos 3t\). 5. Check: at \(t = 0\) it gives about 0.156, close to \(|0| = 0\), and at \(t = \frac{\pi}{2}\) both cosines vanish, giving \(\frac{\pi}{2}\), exactly \(f(\frac{\pi}{2})\).
Measuring the error
The squared distance \(\|f - f_n\|^2 = \int_{-\pi}^{\pi} (f(t) - f_n(t))^2\,dt\) measures the total squared error of the approximation \(f_n\), a continuous version of the sum of squared residuals. Because each \(f_n\) is a projection onto a larger subspace than the one before, this error can never increase as \(n\) grows. For \(|t|\) it falls from about 5.17 with the constant alone, to about 0.075 at first order, to about 0.012 at third order. This is why a few terms often capture a signal well, an idea behind audio and image compression, which use related cosine-type transforms, keep the coefficients that matter most and store the rest coarsely or not at all.
Common mistakes to avoid
First, using \(X^TX\hat{\boldsymbol{\beta}} = X^TW\mathbf{y}\); the weights must appear on both sides. Second, giving the less reliable points the larger weights; a larger weight pulls the fit towards that point. Third, using \(1, t, t^2\) for trend analysis and expecting coefficients to stay fixed when the degree rises; only an orthogonal set has that property. Fourth, forgetting the \(\frac{1}{2}\) on the constant term \(\frac{a_0}{2}\), which comes from \(\langle 1, 1 \rangle = 2\pi\). Fifth, dividing by \(2\pi\) instead of \(\pi\) when computing \(a_k\) or \(b_k\) for \(k \ge 1\).
Key takeaways
Each application is a best approximation in a well-chosen inner product space. Weighted least squares minimises \(\sum w_i(y_i - \hat{y}_i)^2\) and solves \(X^TWX\hat{\boldsymbol{\beta}} = X^TW\mathbf{y}\). Trend analysis at equally spaced, centred times uses orthogonal polynomials such as \(1, t, t^2 - 2\), so each coefficient is one division and higher terms do not disturb lower ones. On \([-\pi, \pi]\), \(1, \cos kt, \sin kt\) are orthogonal, and projecting onto them gives the Fourier approximation with \(a_k = \frac{1}{\pi}\int f\cos kt\,dt\), \(b_k = \frac{1}{\pi}\int f\sin kt\,dt\) and constant term \(\frac{a_0}{2}\). Its error never grows as terms are added.
Flashcards
Flashcard
Weighted least squares, trend analysis, Fourier approximation
Test your understanding
Question 1 of 5
In weighted least squares, what happens when one data point is given a larger weight than the others?
Question 2 of 5
Why does trend analysis use orthogonal polynomials such as 1, t, t² - 2 instead of 1, t, t²?
Question 3 of 5
For f in C[-π, π], which expression gives the Fourier coefficient aₖ for k ≥ 1?
Question 4 of 5
What is the constant term a₀/2 of the Fourier approximation of f(t) = |t| on [-π, π]?
Question 5 of 5
With ⟨f, g⟩ = ∫ f(t)g(t) dt over [-π, π], what is ⟨cos t, cos t⟩?
