Linear Algebra › Section G · Symmetric matrices and quadratic forms › Topic G.3

Constrained optimization

How large can a quadratic form get when its input is restricted to unit vectors, and in which direction is that maximum reached? The answer is an eigenvalue and its eigenvector, and the same idea locates the greatest stress in a beam, the direction of greatest variance in data and the greatest stretch of any matrix.

Why a constraint is needed

A quadratic form has no largest value on all of \(\mathbb{R}^n\) unless it is never positive, because \(Q(t\mathbf{x}) = t^2Q(\mathbf{x})\): doubling the input quadruples the output. The meaningful question is about direction, not size. So we fix the length and ask for the largest and smallest values of \(Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}\) subject to

\|\mathbf{x}\| = 1, \text{ that is, } x_1^2 + \cdots + x_n^2 = 1

The unit vectors form a circle in \(\mathbb{R}^2\) and a sphere in \(\mathbb{R}^3\). Because this set is closed and bounded and \(Q\) is continuous, both a maximum and a minimum always exist; the work is to find them and where they occur.

The diagonal case

Start with no cross terms: maximise and minimise \(Q = 7x_1^2 + 5x_2^2 + 2x_3^2\) on the unit sphere. Each square is non-negative, so replacing the smaller coefficients by 7 can only increase the total:

Q \le 7x_1^2 + 7x_2^2 + 7x_3^2 = 7(x_1^2 + x_2^2 + x_3^2) = 7

The value 7 is reached at \(\mathbf{x} = \pm(1, 0, 0)\), so the maximum is 7. Replacing every coefficient by 2 instead gives \(Q \ge 2\), reached at \(\pm(0, 0, 1)\), so the minimum is 2. The extreme values are the largest and smallest diagonal entries, which here are the eigenvalues, and they occur at the matching standard basis vectors.

The theorem for any symmetric matrix

For a general symmetric \(A\), use the principal axes change of variable \(\mathbf{x} = P\mathbf{y}\) from "Quadratic forms". Since \(P\) is orthogonal, \(\|\mathbf{y}\| = \|\mathbf{x}\|\), so the constraint is unchanged, and \(Q\) becomes \(\lambda_1y_1^2 + \cdots + \lambda_ny_n^2\), the diagonal case. List the eigenvalues as \(\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_n\). Then, subject to \(\|\mathbf{x}\| = 1\):

\max \mathbf{x}^TA\mathbf{x} = \lambda_1, \qquad \min \mathbf{x}^TA\mathbf{x} = \lambda_n

The maximum is attained at a unit eigenvector \(\mathbf{u}_1\) for \(\lambda_1\), and the minimum at a unit eigenvector \(\mathbf{u}_n\) for \(\lambda_n\). Notice that \(\mathbf{u}^TA\mathbf{u} = \lambda\,\mathbf{u}^T\mathbf{u} = \lambda\) for any unit eigenvector, so each eigenvalue is a value of \(Q\) on the sphere.

Worked example: a form with a cross term

Find the extreme values of \(Q = 3x_1^2 + 4x_1x_2 + 6x_2^2\) subject to \(x_1^2 + x_2^2 = 1\), and where they occur. 1. The matrix has rows \((3, 2)\) and \((2, 6)\): trace 9, determinant 14, so the eigenvalues are 7 and 2. 2. For 7, \(A - 7I\) has rows \((-4, 2)\) and \((2, -1)\), giving the unit eigenvector \((1, 2)/\sqrt{5}\). For 2, the eigenvector is \((2, -1)/\sqrt{5}\). 3. The maximum is 7, at \(\pm(1, 2)/\sqrt{5}\); the minimum is 2, at \(\pm(2, -1)/\sqrt{5}\). 4. Check: \(Q\) at \((1, 2)/\sqrt{5}\) is \((3 + 8 + 24)/5 = 7\). For comparison, \(Q(1, 0) = 3\) and \(Q(0, 1) = 6\), both between 2 and 7, as they must be.

The next largest value

Once the top direction \(\mathbf{u}_1\) is known, you can ask for the largest value of \(Q\) among unit vectors perpendicular to it. The answer is \(\lambda_2\), attained at a unit eigenvector \(\mathbf{u}_2\) for \(\lambda_2\). In principal-axis coordinates the extra condition \(\mathbf{x} \cdot \mathbf{u}_1 = 0\) is just \(y_1 = 0\), which removes the largest term and leaves \(\lambda_2\) on top. The pattern continues: subject to \(\|\mathbf{x}\| = 1\) and \(\mathbf{x}\) orthogonal to \(\mathbf{u}_1, \dots, \mathbf{u}_{k-1}\), the maximum of \(\mathbf{x}^TA\mathbf{x}\) is \(\lambda_k\), attained at \(\mathbf{u}_k\). So every eigenvalue, in order, is the answer to a constrained maximum.

Worked example: three variables, step by step

Let \(Q = 3x_1^2 + 3x_2^2 + x_3^2 + 2x_1x_2\). 1. The matrix has rows \((3, 1, 0)\), \((1, 3, 0)\) and \((0, 0, 1)\). The top-left block has eigenvalues 4 and 2, and the last entry adds 1, so \(\lambda_1 = 4\), \(\lambda_2 = 2\), \(\lambda_3 = 1\). 2. Unit eigenvectors: \(\mathbf{u}_1 = (1, 1, 0)/\sqrt{2}\), \(\mathbf{u}_2 = (1, -1, 0)/\sqrt{2}\), \(\mathbf{u}_3 = (0, 0, 1)\). 3. On the unit sphere, the maximum is 4 at \(\pm\mathbf{u}_1\) and the minimum is 1 at \(\pm\mathbf{u}_3\). 4. Among unit vectors with \(x_1 + x_2 = 0\), that is, orthogonal to \(\mathbf{u}_1\), the maximum is 2 at \(\pm\mathbf{u}_2\). 5. Check: \(Q(\mathbf{u}_2) = (3 + 3 - 2)/2 = 2\).

The Rayleigh quotient and Lagrange multipliers

Dividing by the squared length removes the constraint. For every non-zero \(\mathbf{x}\), the Rayleigh quotient satisfies

\lambda_n \le \frac{\mathbf{x}^TA\mathbf{x}}{\mathbf{x}^T\mathbf{x}} \le \lambda_1

with equality at the matching eigenvectors. If you know Lagrange multipliers, they give the same result: setting the gradient of \(\mathbf{x}^TA\mathbf{x}\), which is \(2A\mathbf{x}\), equal to \(\lambda\) times the gradient of \(\mathbf{x}^T\mathbf{x}\), which is \(2\mathbf{x}\), gives \(A\mathbf{x} = \lambda\mathbf{x}\). The candidates are exactly the unit eigenvectors, and the multiplier is the value of \(Q\) there. If the constraint is \(\|\mathbf{x}\| = r\) instead, every value is multiplied by \(r^2\).

Worked example: principal stresses

Inside a loaded steel plate, the stress at a point is described by a symmetric matrix \(\sigma\). The normal stress on a cut whose unit normal is \(\mathbf{n}\) is the quadratic form \(\mathbf{n}^T\sigma\mathbf{n}\). Suppose \(\sigma\) has rows \((50, 30)\) and \((30, -30)\), in megapascals (MPa), with positive meaning tension. 1. Trace 20, determinant \(-1500 - 900 = -2400\), so the eigenvalues solve \(\lambda^2 - 20\lambda - 2400 = 0\): 60 and \(-40\). 2. The greatest normal stress is 60 MPa of tension, on the cut with normal \((3, 1)/\sqrt{10}\); check \(\sigma(3, 1) = (180, 60)\). 3. The least is 40 MPa of compression, with normal \((1, -3)/\sqrt{10}\). Engineers call these the principal stresses, and they compare the largest with the material's strength.

Common mistakes to avoid

First, answering with the largest diagonal entry: for rows \((3, 2)\) and \((2, 6)\) the maximum is 7, not 6, because the cross term matters. Second, giving an eigenvector that is not a unit vector as the place of the maximum; \((1, 2)\) has length \(\sqrt{5}\), and \(Q(1, 2) = 35\), not 7. Third, confusing the maximum value, an eigenvalue, with where it occurs, an eigenvector. Fourth, forgetting the \(r^2\) factor when the constraint is \(\|\mathbf{x}\| = r\) rather than 1. Fifth, forgetting the extra orthogonality condition: \(\lambda_2\) is the largest value only among unit vectors perpendicular to \(\mathbf{u}_1\).

Key takeaways

Without a constraint, a quadratic form is unbounded unless it is never positive, since Q(tx) = t²Q(x). Subject to ‖x‖ = 1, the maximum of xᵀAx is the largest eigenvalue λ₁ of the symmetric matrix A, reached at a unit eigenvector u₁, and the minimum is the smallest eigenvalue, reached at its unit eigenvector. Adding the condition x ⊥ u₁ makes the maximum λ₂, and in general orthogonality to u₁, …, uₖ₋₁ gives λₖ. Equivalently, the Rayleigh quotient xᵀAx / xᵀx lies between the smallest and largest eigenvalues, and Lagrange multipliers lead to the eigenvalue equation Ax = λx. Principal stresses are one engineering use.

Flashcards

Flashcard

Q(tx)

t²Q(x)

Why a length constraint is needed.

Test your understanding

Question 1 of 5

What is the maximum of Q = 2x₁² + 6x₁x₂ + 2x₂² subject to x₁² + x₂² = 1?

Question 2 of 5

A symmetric 3 × 3 matrix has eigenvalues 9, 4 and −1 with unit eigenvectors u₁, u₂, u₃. Among unit vectors x orthogonal to u₁, what is the largest value of xᵀAx?

Question 3 of 5

For the matrix with rows (3, 2) and (2, 6), where on the unit circle does Q = xᵀAx reach its minimum?

Question 4 of 5

A symmetric matrix A has eigenvalues 5 and 1. What is the maximum of xᵀAx subject to ‖x‖ = 3?

Question 5 of 5

The stress matrix at a point has rows (40, 0) and (0, −20) MPa. Which statement about the normal stress nᵀσn on cuts with unit normal n is true?

← Previous topicNext topic →