🧮 Gradient Descent and Backpropagation Training Lab

About this simulation

Training a neural network means adjusting its weights to make a loss function smaller. Backpropagation computes, for every weight, the gradient — the direction that most increases the loss — by applying the chain rule backward from the output. Gradient descent then moves the weights a small step in the opposite direction. This simulation shrinks that whole process down to just two weights, so the loss function becomes a landscape you can see and the optimizer becomes a marker rolling across it.

How it works

• The surface height at any point is the loss for that pair of weight values; the marker's world position is literally (w1, loss, w2).
• At every step the true analytic gradient of the surface is computed — the same partial derivatives backpropagation would compute for a real network — and drawn as an arrow pointing downhill.
• Gradient descent steps straight along the negative gradient, so on a steep, narrow ravine it zig-zags across the walls instead of moving efficiently toward the floor.
• Momentum keeps a running average of past gradients, which cancels the zig-zag and lets the marker build up speed along the ravine's floor.
• Adam additionally rescales each direction by its own recent gradient magnitude, so it takes small careful steps where the surface is steep and larger steps where it is flat.

Controls

• Optimizer — switch between vanilla gradient descent, momentum, and Adam.
• Loss surface — a symmetric bowl, an elongated ravine that exposes zig-zagging, or a bumpy surface with a misleading local minimum.
• Learning rate — the step size η applied to every update; too high and the marker can overshoot or diverge.
• Momentum (β) — how strongly past gradients are remembered, used by both the Momentum optimizer and as Adam's β₁.
• ⏸ Pause / ↺ New start point — pause training or restart the marker from a random point on the landscape.

Did you know?

Adam (Adaptive Moment Estimation), introduced by Kingma & Ba in 2015, combines momentum with per-parameter adaptive learning rates and is the default optimizer for the vast majority of modern deep learning training runs.