The bumpy bowl is a synthetic loss surface (a Rosenbrock-like valley plus
tunable noise ripples). Each algorithm reads the local gradient and takes a step
— the ball's trail shows the actual optimization path. Plain gradient descent can
oscillate across a steep, narrow valley; momentum smooths that out by carrying
velocity forward; the Adam-style variant additionally normalizes each step by a
running estimate of gradient magnitude, so it adapts its effective step size
per-direction.
Plain GD: θ ← θ − η·∇L(θ)
Momentum: v ← β·v − η·∇L(θ); θ ← θ + v
Adam-style: m ← β1·m+(1−β1)·g; s ← β2·s+(1−β2)·g²
θ ← θ − η · m/(√s + ε)
- Learning rate — step size η; too high overshoots the minimum, too low converges slowly.
- Momentum — how much of the previous velocity carries forward, damping oscillation in narrow valleys.
- Surface roughness — added high-frequency noise on the loss surface, the way real non-convex loss landscapes have local wiggles that can trap a naive optimizer.
This is exactly the algorithmic refinement a physics/simulation engine relies
on when it fits parameters or tunes a solver — the same three-way trade-off
between speed, stability and convergence applies whether it's ML model weights
or a simulation's numerical solver settings.