Heavy-ball momentum on a narrow valley (3D)

Two balls roll down the same quadratic valley: plain gradient descent at its best step, and a heavy ball with momentum. Watch the convergence rate follow an exact formula.

Parameters

Camera

Quality

Model checks

The science

What is modelled

The loss is the quadratic bowl f(x, y) = ½(μ x² + L y²) with L = 1 and μ = 1/κ. A long, narrow valley means a large condition number κ. The steel ball uses plain gradient descent with the best constant step α = 2/(L+μ); the copper ball uses the heavy-ball method xk+1 = xk - α ∇f(xk) + β (xk - xk-1) with the step and momentum you choose.

Equations

  • The Hessian is diagonal, so each direction with curvature λ evolves independently: ek+1 = (1 + β - αλ) ek - β ek-1.
  • Its characteristic polynomial is z² - (1 + β - αλ) z + β = 0. The convergence factor is the spectral radius ρ = max |z|: if the discriminant is negative both roots are complex with |z| = √β, otherwise ρ is the larger real root in magnitude.
  • Plain gradient descent: ρ = max(|1 - αμ|, |1 - αL|), minimal at α = 2/(L+μ) giving (κ - 1)/(κ + 1).
  • Polyak’s optimal heavy ball for a quadratic: α = 4/(√L + √μ)², β = ((√κ - 1)/(√κ + 1))², giving ρ = (√κ - 1)/(√κ + 1). The number of iterations to a fixed accuracy grows like κ for gradient descent but only like √κ with momentum.
  • Stability needs 0 ≤ β < 1 and α < 2(1+β)/L.

Checks run in this page

  • Recurrence against roots: the scalar recurrence is iterated for both curvatures and its measured envelope decay rate is compared with the analytic spectral radius (within 3 %).
  • Closed forms: the numerical spectral radius at the optimal gradient-descent step equals (κ-1)/(κ+1), and at the Polyak parameters equals (√κ-1)/(√κ+1) in both directions.
  • Stability boundary: just below α = 2(1+β) the iteration decays, just above it grows.

Honest simplifications

  • Exact only for quadratic losses. Real neural-network losses are not quadratic; momentum, Adam and learning-rate schedules behave differently, and heavy-ball can fail to converge on some non-quadratic convex functions. This is the textbook local model near a minimum.
  • The ball is not a physical rolling ball: it jumps along discrete iterates (the animation interpolates between them). Gradients are exact, with no minibatch noise.
  • Surface height is the loss multiplied by a display factor so the valley fits the picture; the axes are not to scale.

The balls, plinth, flag and baked stone texture are Blender-built; the optimiser is computed live in your browser.

Loading model...
Loading Blender model...
Error ||x_k|| against iteration (log); dashed = e0 rho^k
Spectral radius against momentum beta
Iterations to 1e-6 against kappa
Made with 🟠 Blender · Three.js