Gradient Descent and Backpropagation: How Neural Networks Actually Learn

Learning as Descending a Loss Landscape

A neural network's weights start out as small random numbers, so its earliest predictions are essentially noise. Training is the process of nudging those weights, prediction by prediction, until the network's outputs consistently match the correct answers. To do that systematically, you need two things: a single number that measures how wrong the current predictions are, called the loss, and a rule for adjusting every weight so that the loss gets smaller.

It helps to picture the loss as a landscape, with every possible combination of weight values corresponding to one point on that landscape's surface, and the height at that point equal to the loss achieved by those weights. Training a network is equivalent to starting at some random point on this landscape (the initial random weights) and trying to walk downhill toward a valley (a set of weights with low loss). The landscape has far more dimensions than we can visualise — a modest network can easily have millions of weights, meaning millions of dimensions — but the intuition of "figure out which direction is downhill from here, and take a step" carries over exactly.

The direction that most steeply increases the loss, at any given point on this landscape, is given by the gradient — the vector of partial derivatives of the loss with respect to every weight. Since we want to decrease the loss, not increase it, we step in the exact opposite direction of the gradient. This is gradient descent: repeatedly compute the gradient, take a small step opposite to it, and repeat until the loss stops improving meaningfully.

Backpropagation: Computing the Gradient Efficiently

Knowing that we need the gradient is one thing; computing it efficiently for a network with millions of weights spread across many layers is another. This is the problem backpropagation solves, using the chain rule from calculus to break a seemingly intractable computation into a sequence of manageable local steps.

Training proceeds in two passes. The forward pass feeds an input through the network layer by layer — each layer computes a weighted sum of its inputs (Z = W·A + b), then applies a non-linear activation function (A = activation(Z)) — until the final layer produces a prediction, which is compared against the true label to compute the loss.

The backward pass then works in reverse, starting from the loss and propagating gradient information back toward the input, one layer at a time. At the output layer, when using softmax activation with cross-entropy loss (the standard combination for classification), the gradient with respect to the pre-activation values simplifies elegantly to dZ = predicted − true — literally the difference between what the network predicted and what it should have predicted. From there, the chain rule lets each layer compute its own weight gradients using only the gradient signal handed to it by the layer after it, plus its own locally-stored activations from the forward pass:

dW[layer] = (1/m) · A[layer-1]ᵀ · dZ[layer]
db[layer] = (1/m) · sum(dZ[layer])
dA[layer-1] = dZ[layer] · W[layer]ᵀ
dZ[layer-1] = dA[layer-1] ⊙ activation'(Z[layer-1])

where m is the batch size and ⊙ denotes element-wise multiplication. The crucial efficiency insight is that each layer's gradient calculation reuses the gradient already computed for the layer downstream of it, rather than recomputing everything from scratch — this is what makes backpropagation's total cost roughly proportional to a single forward pass, instead of exploding combinatorially with network depth, which is the property that makes training deep networks computationally feasible at all.

Developers commonly verify a hand-written backpropagation implementation with gradient checking: nudging one weight up by a tiny epsilon, rerunning the forward pass, nudging it down by the same epsilon, rerunning again, and comparing the resulting numerical slope, (loss_plus − loss_minus) / (2·epsilon), against the analytical gradient backpropagation produced. Close agreement (typically within 1e-7) confirms the backward pass is correctly implemented — an important sanity check, since a subtle bug in gradient calculation often still trains "sort of," just badly, which makes it easy to miss without an explicit numerical check.

Three Flavours of Gradient Descent

Once you can compute a gradient, you still have to decide how much data to use for each gradient calculation before updating the weights, and this choice matters a great deal in practice. Batch gradient descent computes the gradient using the entire training set before taking a single step. This gives a very accurate, low-noise estimate of the true downhill direction, but for a dataset with millions of examples, a single weight update becomes prohibitively slow, and the memory required to hold the whole dataset's activations at once can be impractical.

Stochastic gradient descent (SGD) goes to the opposite extreme, computing the gradient and updating weights after every single training example. This is fast per-step and the noisiness of the resulting updates can actually help the optimiser escape shallow local minima or saddle points, but the path toward the minimum becomes erratic, and per-example updates don't take advantage of the parallel hardware (GPUs) that make batched matrix operations efficient.

Mini-batch gradient descent is the practical compromise nearly everyone uses: compute the gradient over a modest batch of examples (commonly 32, 64 or 128) before each update. This captures most of the stability of full-batch gradients while remaining fast enough to make many updates per epoch, and batch sizes in this range map efficiently onto GPU matrix operations.

Momentum, RMSprop and Adam: Smarter Step-Taking

Plain gradient descent takes every step at a fixed size and direction determined purely by the current gradient, which turns out to be inefficient on real loss landscapes — narrow ravines cause it to oscillate back and forth across the ravine walls while making frustratingly slow progress along the ravine's length. Modern optimisers address this by remembering information from previous steps.

Momentum keeps a running, exponentially-decayed average of past gradients (a "velocity" term) and updates weights using that average rather than the instantaneous gradient alone: v = β·v + (1−β)·gradient, then weights −= learning_rate·v, typically with β around 0.9. Gradients that consistently point the same way across several steps reinforce each other and accelerate progress in that direction, while gradients that flip sign from step to step (the oscillation across a ravine) partially cancel out, damping the oscillation.

RMSprop takes a different tack, keeping a running average of squared gradients per weight and dividing the learning rate by the square root of that average: s = β·s + (1−β)·gradient², then weights −= learning_rate·gradient / sqrt(s + ε). This effectively gives each individual weight its own adaptive learning rate — weights with consistently large gradients get their effective step size shrunk, while weights with small, infrequent gradients get relatively larger steps, which helps when different weights need very different update magnitudes.

Adam (Adaptive Moment Estimation) combines both ideas: it tracks momentum (called the first moment, m) and the RMSprop-style squared-gradient average (the second moment, v) simultaneously, applies a bias correction to each (important in early training steps, when both moving averages start at zero and are biased toward zero), and then updates weights using the corrected momentum divided by the square root of the corrected squared-gradient average. With its commonly recommended defaults (β₁=0.9, β₂=0.999, ε=1e-8), Adam converges quickly with relatively little manual tuning across a wide range of problems, which is why it has become the default first choice for training most modern neural networks, including nearly all convolutional and transformer-based computer vision models.

Learning Rate: The Single Most Consequential Hyperparameter

Every one of these optimisers still needs a learning rate — the size of the step taken relative to the (possibly adapted) gradient direction — and this single number has an outsized effect on whether training succeeds at all. A learning rate that is too large causes updates to repeatedly overshoot the minimum, and in the worst case can make the loss diverge to infinity instead of decreasing. A learning rate that is too small makes training technically stable but excruciatingly slow, and can leave the optimiser stuck for a very long time in a flat or slightly bumpy region of the loss landscape that a larger step would have escaped easily.

Rather than fixing a single learning rate for the entire training run, most practical training schedules reduce it over time following one of several patterns: step decay drops the rate by a fixed factor every few epochs; exponential decay shrinks it continuously by a constant ratio each epoch; cosine annealing smoothly reduces it following a cosine curve down to near zero by the end of training. The underlying logic in every case is the same: use a relatively large learning rate early on, when the weights are far from any good solution and large steps make fast progress, and use a smaller learning rate later, when the weights are close to a good region and large steps would just cause oscillation around the minimum rather than settling into it.