Training a neural network means repeatedly nudging its weights downhill on a loss surface — a landscape where height represents error. The learning rate controls the size of every step: too small and training crawls or gets trapped in a shallow dip; too large and the optimizer overshoots, bounces off the walls of a valley, or diverges outright. This sim renders that landscape in 3D and drops a ball onto it that follows real gradient descent with momentum, so you can watch the trade-off happen instead of just reading about it.
position -= lr × gradient + momentum × previous_step, exactly like
SGD with momentum.
A learning rate that's an order of magnitude too high is one of the most common
reasons a neural network's loss explodes to NaN in the first few
steps of training — the "ball" flies off the surface entirely, which is exactly
what you'll see if you push η near its maximum here on the ravine surface.
A ball rolls across a 3D loss surface under real gradient descent with momentum, letting you feel how learning rate and schedule choices trade off convergence speed against oscillation and divergence.
Each step computes the local gradient and updates position by learning rate × slope plus momentum-carried velocity — the same update rule used in SGD, so overshoot, oscillation, and stalling behave exactly as they do in real training.
Set the learning rate and momentum, pick a schedule and a loss surface, then watch the loss chart. Push the learning rate high on the ravine surface to see divergence; keep it low on the bumpy surface to see it stall in a local minimum.
Cosine annealing and step decay were popularized because a large learning rate helps escape shallow local minima early in training, while a small one late in training lets the optimizer settle precisely into a deep minimum.