Presets
Playback
Valley shape
Ratio of the steepest to the flattest curvature.
Optimiser settings
Step size and momentum are shared by the three methods.
Time
How many optimiser iterations happen each second.
Layers
Model checks
The science
The valley
The loss is the quadratic f(x, y) = (kappa x^2 + y^2) / 2. Its Hessian has curvatures L = kappa (steep, across the valley) and mu = 1 (flat, along it), so the condition number is kappa = L / mu. The gradient is (kappa x, y). The 3D surface shows height sqrt(f), so the walls keep the same slope for every kappa while the floor flattens.
Three optimisers
Gradient descent: x ← x - alpha grad f(x). Heavy ball (Polyak): v ← beta v - alpha grad f(x), x ← x + v. Nesterov: v ← beta v - alpha grad f(x + beta v), x ← x + v, so the gradient is taken at the look-ahead point. The step is alpha = c / L.
Exact rates
On a quadratic every direction decouples. With h = alpha lambda each coordinate obeys x(k+1) = a x(k) - b x(k-1): gradient descent has (a, b) = (1 - h, 0), heavy ball (1 + beta - h, beta), Nesterov ((1 - h)(1 + beta), beta (1 - h)). The convergence rate is the spectral radius of the companion matrix: sqrt(b) when the roots are complex (oscillating, momentum-dominated) and (|a| + sqrt(a^2 - 4b)) / 2 when they are real. The slowest of the two modes decides the speed. Gradient descent needs about kappa ln(1/eps) iterations, momentum methods with tuned beta about sqrt(kappa) ln(1/eps); the best beta for Nesterov at c = 1 is (sqrt(kappa) - 1) / (sqrt(kappa) + 1).
Simplifications
Neural-network losses are not quadratic and have stochastic gradients, so the exact rates here are an idealisation; the qualitative message (momentum turns kappa into sqrt(kappa), too much beta overshoots) carries over. Nesterov's optimal-rate guarantee for non-quadratic convex functions uses a time-varying beta, not the constant used here.
What to try
- Keep kappa = 100 and compare the iterations needed: gradient descent about 1400, momentum methods with beta near 0.8 about 100-200.
- Set beta = 0: all three methods coincide.
- Press the heavy-momentum preset and watch the balls overshoot back and forth across the valley and the loss graph oscillate.
- Press the aggressive-step preset (1.9 / L): plain gradient descent still converges, momentum with the wrong beta can diverge.
- Move kappa from 1 to 1000 in graph 3 and see the gap between the gradient-descent line and the momentum lines widen.