Presets
Parameters
Model checks
The science
What is modelled
The loss is the quadratic bowl f(x, y) = ½(μ x² + L y²) with L = 1 and μ = 1/κ. A long, narrow valley means a large condition number κ. The steel ball uses plain gradient descent with the best constant step α = 2/(L+μ); the copper ball uses the heavy-ball method xk+1 = xk - α ∇f(xk) + β (xk - xk-1) with the step and momentum you choose.
Equations
- The Hessian is diagonal, so each direction with curvature
λevolves independently:ek+1 = (1 + β - αλ) ek - β ek-1. - Its characteristic polynomial is
z² - (1 + β - αλ) z + β = 0. The convergence factor is the spectral radiusρ = max |z|: if the discriminant is negative both roots are complex with|z| = √β, otherwiseρis the larger real root in magnitude. - Plain gradient descent:
ρ = max(|1 - αμ|, |1 - αL|), minimal atα = 2/(L+μ)giving(κ - 1)/(κ + 1). - Polyak’s optimal heavy ball for a quadratic:
α = 4/(√L + √μ)²,β = ((√κ - 1)/(√κ + 1))², givingρ = (√κ - 1)/(√κ + 1). The number of iterations to a fixed accuracy grows likeκfor gradient descent but only like√κwith momentum. - Stability needs
0 ≤ β < 1andα < 2(1+β)/L.
Checks run in this page
- Recurrence against roots: the scalar recurrence is iterated for both curvatures and its measured envelope decay rate is compared with the analytic spectral radius (within 3 %).
- Closed forms: the numerical spectral radius at the optimal gradient-descent step equals
(κ-1)/(κ+1), and at the Polyak parameters equals(√κ-1)/(√κ+1)in both directions. - Stability boundary: just below
α = 2(1+β)the iteration decays, just above it grows.
Honest simplifications
- Exact only for quadratic losses. Real neural-network losses are not quadratic; momentum, Adam and learning-rate schedules behave differently, and heavy-ball can fail to converge on some non-quadratic convex functions. This is the textbook local model near a minimum.
- The ball is not a physical rolling ball: it jumps along discrete iterates (the animation interpolates between them). Gradients are exact, with no minibatch noise.
- Surface height is the loss multiplied by a display factor so the valley fits the picture; the axes are not to scale.
What the 2D view adds
- Level curves
f = care ellipses with semi-axes√(2c/μ)and√(2c); their elongation is√κ. Plain gradient descent zig-zags across the narrow direction, momentum smooths the path. - Tap or drag on the map to move the starting point; the convergence rate does not depend on it, only the number of iterations does.
The ball, flag and stone-plinth sprites are Blender 5.2 renders of the same models used in the 3D version; the optimiser is computed live in your browser on one shared maths module (hb_core.js).
Loading sprites...