Gradient Descent & Modern Optimisers: Adam, RMSprop, Momentum

Following the slope

Picture the model's error as a landscape of hills and valleys, where the horizontal coordinates are the parameters and height is the loss. The gradient points toward steepest increase, so gradient descent moves in the opposite direction:

θ = θ − η · ∇J(θ)

θ = parameters, η = learning rate, ∇J(θ) = gradient of loss

Set η too high and steps overshoot the valley, bouncing or diverging; set it too low and training crawls. Computing the gradient over the whole dataset is slow, so practice uses mini-batch gradient descent — estimating the gradient from a small random sample (commonly 32-512 examples) each step, which is both fast and injects helpful noise that can shake the optimiser out of poor regions.

live demo · SGD, Momentum, RMSprop and Adam racing down a 3D loss landscape● LIVE

Momentum, RMSprop and Adam

Plain SGD struggles on long, narrow ravines, zig-zagging across the walls while crawling along the floor. Momentum accumulates an exponentially weighted velocity from past gradients (v = β·v + ∇J(θ), typically β≈0.9), so consistent directions build speed and oscillations cancel. RMSprop instead divides each step by a running root-mean-square of recent gradient magnitudes, giving every parameter its own adaptive learning rate — damping noisy parameters, amplifying quiet ones. Adam combines both: a momentum-like first moment plus an RMSprop-like second moment, with bias correction for early steps. With sensible defaults (β₁=0.9, β₂=0.999, ε=1e-8) it often trains reliably with minimal tuning, which is why it became deep learning's default workhorse — though well-tuned SGD with momentum sometimes generalises better, particularly in computer vision.

Saddle points, not local minima, are the real obstacle

A common misconception is that gradient descent finds the single global minimum — in practice, especially for neural networks, it finds a good-enough minimum, and that's usually sufficient. Another is that local minima are the main danger; in the very high-dimensional loss landscapes of real networks, true local minima are statistically rare, and saddle points — where the gradient vanishes but the surface curves up in some directions and down in others — are the bigger obstacle, since plain gradient descent can stall there. Momentum and adaptive methods help carry the optimiser through.

Frequently asked questions

What is gradient descent in simple terms?

Gradient descent is an iterative method that repeatedly nudges a model's parameters in the direction that most reduces an error (loss) function. By following the downhill slope of that function, it gradually settles into a configuration that makes accurate predictions.

What does momentum actually do?

Momentum accumulates an exponentially weighted average of past gradients, so the update gains velocity along consistent directions and dampens oscillations across narrow valleys. This typically speeds up convergence and helps the optimiser glide through shallow, flat regions of the loss surface.

Why is Adam so popular?

Adam combines momentum's smoothed direction with RMSprop's per-parameter adaptive scaling, plus bias correction for early steps. It often trains reliably with minimal tuning, which is why it became a default workhorse across deep learning — though well-tuned SGD with momentum sometimes generalises better.

Try it live

Everything above runs in your browser — open Gradient Descent Visualiser and watch SGD, Momentum, RMSprop and Adam navigate a 3D loss landscape, comparing convergence speed and trajectory shape live.

▶ Open Gradient Descent simulation