The Gradient Points Uphill
The gradient packages all the partial derivatives into one vector. It points in the direction of steepest increase, and its dot product with any direction gives the rate of change that way.
Beyond the axis directions
Partial derivatives tell you the slope of a surface heading due east (\(x\)-direction) or due north (\(y\)-direction). But a hiker can walk in any direction. The directional derivative measures the rate of change of \(f\) in the direction of any unit vector \(\mathbf{u}\). It turns out to be built entirely from the partial derivatives.
The gradient vector
The gradient of \(f(x, y)\) is the vector of its partial derivatives, written \(\nabla f\) and read 'del f' or 'grad f'. In three variables, add \(f_z\). For \(f = x^2 + 3xy\), \(\nabla f = \langle 2x + 3y,\ 3x\rangle\), and at \((1, 2)\) it is \(\langle 8, 3\rangle\).
\nabla f = \langle f_x, f_y \rangleDirectional derivatives
For a unit vector \(\mathbf{u}\), the directional derivative is the dot product of the gradient with \(\mathbf{u}\). If \(\mathbf{u}\) is not a unit vector, divide it by its length first; otherwise the answer is scaled incorrectly. With \(\nabla f = \langle 8, 3\rangle\) and \(\mathbf{u} = \langle \frac{3}{5}, \frac{4}{5}\rangle\): \(D_\mathbf{u}f = \frac{24}{5} + \frac{12}{5} = \frac{36}{5}\).
D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}Steepest ascent
Since \(D_\mathbf{u}f = |\nabla f||\mathbf{u}|\cos\theta = |\nabla f|\cos\theta\), the rate is largest when \(\theta = 0\): moving in the direction of the gradient. So the gradient points in the direction of steepest increase, and its magnitude is that steepest rate. Moving opposite to the gradient gives the steepest decrease, the idea behind gradient descent, the algorithm that trains most machine learning models.
\max D_{\mathbf{u}}f = |\nabla f|The gradient is perpendicular to level curves
Along a level curve, \(f\) does not change, so the directional derivative along it is \(0\): the gradient is perpendicular to the level curve. On a contour map, the steepest uphill path crosses the contour lines at right angles. In three variables, \(\nabla f\) is perpendicular to level surfaces, which gives normal vectors for tangent planes to surfaces like spheres.
Worked example: a hill
A hill has height \(h(x, y) = 100 - x^2 - 2y^2\) metres. At \((3, 1)\), \(\nabla h = \langle -2x, -4y\rangle = \langle -6, -4\rangle\). The steepest climb is in the direction \(\langle -6, -4\rangle\), back toward the summit at the origin, with slope \(|\nabla h| = \sqrt{52} \approx 7.2\). Walking in the direction \(\langle 4, -6\rangle\), perpendicular to the gradient, keeps your height constant.
Tangent planes to level surfaces
For a surface given by \(F(x, y, z) = k\), the gradient \(\nabla F\) at a point is normal to the surface. So the tangent plane at \((a, b, c)\) is \(F_x(x - a) + F_y(y - b) + F_z(z - c) = 0\). For the sphere \(x^2 + y^2 + z^2 = 14\) at \((1, 2, 3)\), the normal is \(\langle 2, 4, 6\rangle\), giving the plane \(x + 2y + 3z = 14\).
Lagrange multipliers
To maximise \(f(x, y)\) subject to a constraint \(g(x, y) = k\), look for points where the gradients are parallel: \(\nabla f = \lambda\nabla g\). At the optimum, the level curve of \(f\) just touches the constraint curve. For example, maximising \(xy\) on \(x + y = 10\) gives \(\langle y, x\rangle = \lambda\langle 1, 1\rangle\), so \(x = y = 5\), matching the square from single-variable optimization.
\nabla f = \lambda \nabla gCommon mistakes
Using a non-unit direction vector in the directional derivative. Confusing the gradient (a vector) with the maximum rate (its magnitude, a number). Forgetting that the gradient is evaluated at a point. Thinking the gradient lies along a level curve; it is perpendicular to it.
Try it yourself
For \(f(x, y) = x^2y\) at \((2, 1)\), find the gradient and the maximum rate of increase. Answer: \(\nabla f = \langle 2xy, x^2\rangle = \langle 4, 4\rangle\); the maximum rate is \(\sqrt{32} = 4\sqrt{2}\). Then find the directional derivative toward \(\langle 1, 0\rangle\): it is just \(f_x = 4\).
Key takeaways
\(\nabla f = \langle f_x, f_y\rangle\). Directional derivative: \(\nabla f\cdot\mathbf{u}\) with \(\mathbf{u}\) a unit vector. The gradient points in the steepest uphill direction with rate \(|\nabla f|\), and is perpendicular to level curves and surfaces. Lagrange multipliers solve constrained optimization with \(\nabla f = \lambda\nabla g\).
Where gradients appear
Gradients describe heat flowing from hot to cold, water running downhill, electric fields pointing away from charges, and the 'downhill' steps that train neural networks. Whenever a quantity varies over space, its gradient tells you which way it changes fastest and how fast.
Flashcard
Gradient
Check your understanding
At a point, grad f = <6, 8>. What is the maximum rate of increase of f there?
