Heavy-ball momentum on a contour map (2D)

The same narrow valley seen from above: level curves of the loss, the zig-zag of plain gradient descent and the smoother path of the heavy ball. Tap the map to move the start.

Switch to 3D

Presets

Parameters

Model checks

The science

What is modelled

The loss is the quadratic bowl f(x, y) = ½(μ x² + L y²) with L = 1 and μ = 1/κ. A long, narrow valley means a large condition number κ. The steel ball uses plain gradient descent with the best constant step α = 2/(L+μ); the copper ball uses the heavy-ball method xk+1 = xk - α ∇f(xk) + β (xk - xk-1) with the step and momentum you choose.

Equations

  • The Hessian is diagonal, so each direction with curvature λ evolves independently: ek+1 = (1 + β - αλ) ek - β ek-1.
  • Its characteristic polynomial is z² - (1 + β - αλ) z + β = 0. The convergence factor is the spectral radius ρ = max |z|: if the discriminant is negative both roots are complex with |z| = √β, otherwise ρ is the larger real root in magnitude.
  • Plain gradient descent: ρ = max(|1 - αμ|, |1 - αL|), minimal at α = 2/(L+μ) giving (κ - 1)/(κ + 1).
  • Polyak’s optimal heavy ball for a quadratic: α = 4/(√L + √μ)², β = ((√κ - 1)/(√κ + 1))², giving ρ = (√κ - 1)/(√κ + 1). The number of iterations to a fixed accuracy grows like κ for gradient descent but only like √κ with momentum.
  • Stability needs 0 ≤ β < 1 and α < 2(1+β)/L.

Checks run in this page

  • Recurrence against roots: the scalar recurrence is iterated for both curvatures and its measured envelope decay rate is compared with the analytic spectral radius (within 3 %).
  • Closed forms: the numerical spectral radius at the optimal gradient-descent step equals (κ-1)/(κ+1), and at the Polyak parameters equals (√κ-1)/(√κ+1) in both directions.
  • Stability boundary: just below α = 2(1+β) the iteration decays, just above it grows.

Honest simplifications

  • Exact only for quadratic losses. Real neural-network losses are not quadratic; momentum, Adam and learning-rate schedules behave differently, and heavy-ball can fail to converge on some non-quadratic convex functions. This is the textbook local model near a minimum.
  • The ball is not a physical rolling ball: it jumps along discrete iterates (the animation interpolates between them). Gradients are exact, with no minibatch noise.
  • Surface height is the loss multiplied by a display factor so the valley fits the picture; the axes are not to scale.

What the 2D view adds

  • Level curves f = c are ellipses with semi-axes √(2c/μ) and √(2c); their elongation is √κ. Plain gradient descent zig-zags across the narrow direction, momentum smooths the path.
  • Tap or drag on the map to move the starting point; the convergence rate does not depend on it, only the number of iterations does.

The ball, flag and stone-plinth sprites are Blender 5.2 renders of the same models used in the 3D version; the optimiser is computed live in your browser on one shared maths module (hb_core.js).

Loading sprites...
Error against iteration (log); dashed = e0 rho^k
Spectral radius against momentum beta
Iterations to 1e-6 against kappa
Made with 🟠 Blender