Gradient Descent: Every Training Run Is a Ball Rolling Downhill

The basic move: step opposite the gradient

Gradient descent computes the gradient of the loss at the current position — the direction of steepest increase — and moves a little in the opposite direction, downhill. Repeat until the loss stops meaningfully decreasing. This one mechanic, scaled up to millions of dimensions and computed via backpropagation, is what trains nearly every model on this site.

Why real loss surfaces are bumpy, not smooth bowls

Real neural network loss landscapes are highly non-convex, with many local structures rather than one clean global minimum. On a bumpy surface, where gradient descent ends up depends heavily on where it starts — different starting points can descend into entirely different basins, reaching different final loss values, which is exactly why training runs with different random initializations can land at meaningfully different performance.

Learning rate: the difference between crawling and flying off the map

Learning rate controls step size. Too small, and descent crawls, needing many steps to make real progress. Too large, and a step can overshoot the minimum entirely, landing somewhere with an even steeper gradient pointing back — and if each subsequent step gets larger instead of smaller, the result is visible divergence, not just slow convergence.

What momentum changes

Momentum accumulates a running average of past gradients rather than reacting only to the current one, the way a rolling ball carries velocity rather than instantly redirecting at every bump. This smooths out oscillation across narrow valleys and can carry the optimizer through small local bumps that would otherwise stall plain gradient descent — at some cost of occasionally overshooting a minimum it would have stopped at cleanly without momentum.