Backpropagation: How a Neural Network Actually Learns

A network is a function you can differentiate

Strip away the biological metaphor and a feed-forward neural network is a long chain of two alternating operations: an affine map (multiply by a weight matrix, add a bias) and a non-linearity applied element-wise. A network with two hidden layers is nothing more than

a⁰ = x                                  // the input
z¹ = W¹ a⁰ + b¹      a¹ = φ(z¹)          // hidden layer 1
z² = W² a¹ + b²      a² = φ(z²)          // hidden layer 2
z³ = W³ a² + b³      ŷ  = σ(z³)          // output
L  = loss(ŷ, y)                          // a single number

The non-linearity is not decoration. Remove it and the whole stack collapses: a composition of affine maps is one affine map, and a hundred layers would have exactly the expressive power of one. It is φ that buys you everything.

Training means finding the weights that make L small. L is a scalar function of a few thousand or a few billion parameters, so the natural tool is gradient descent — nudge every parameter a little way down its own partial derivative of L. The only question is how to obtain those millions of partial derivatives without doing millions of separate computations. That is what backpropagation answers.

The chain rule, applied in the economical direction

You could get ∂L/∂w by finite differences: perturb w, re-run the network, see how L changed. That costs one full forward pass per parameter — hopelessly expensive, and numerically noisy. Backpropagation instead applies the chain rule from the loss backwards, and computes every gradient in the network in a single backward sweep whose cost is roughly the same as one forward pass.

The trick is to propagate one intermediate quantity, conventionally written δ — the sensitivity of the loss to a layer's pre-activation z:

δᴸ = ∂L/∂zᴸ = ∇_ŷ L ⊙ σ'(zᴸ)              // output layer

δˡ = (Wˡ⁺¹ᵀ δˡ⁺¹) ⊙ φ'(zˡ)                 // every earlier layer

∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ                        // gradient of the weights
∂L/∂bˡ = δˡ                                // gradient of the biases

Read the middle line carefully, because it is the whole algorithm: to get the error signal at a layer, take the error signal from the layer above, push it back through the transpose of that layer's weight matrix, and modulate it by the derivative of this layer's activation. The forward pass sends activations up through W; the backward pass sends errors down through Wᵀ. Each weight's gradient is then just an outer product of the error arriving at its output and the activation that entered its input.

Two consequences follow immediately. First, you must cache the activations from the forward pass — the backward pass needs every aˡ⁻¹ and every zˡ. That is why training a deep network costs so much more memory than running it. Second, backpropagation is not a learning algorithm; it is a gradient computation. The learning is done by whatever optimiser consumes the gradients.

Activations, and the gradient that dies

The sigmoid, φ(z) = 1/(1 + e^(−z)), was the classical choice, and its derivative is the reason deep networks were considered untrainable for years. That derivative is φ(z)(1 − φ(z)), whose maximum value is 0.25, at z = 0, and which falls to nearly zero whenever the neuron saturates. Each layer of backpropagation multiplies the error signal by one of those factors — so through ten sigmoid layers the gradient is scaled by at most 0.25¹⁰ ≈ 10⁻⁶ even in the best case. This is the vanishing gradient problem: the early layers receive essentially no signal and never learn.

φ(z) = 1/(1+e^-z)     φ'(z) = φ(1−φ)      ∈ (0, 0.25]   saturates
φ(z) = tanh(z)        φ'(z) = 1 − tanh²z   ∈ (0, 1]      zero-centred
φ(z) = max(0, z)      φ'(z) = 1 if z > 0   ∈ {0, 1}      ReLU
                                 else 0                  can die
φ(z) = max(αz, z)     φ'(z) = 1 or α       leaky ReLU, α ≈ 0.01

ReLU fixes it by brute force: for any positive input its derivative is exactly 1, so the error signal passes through undiminished no matter how many layers it crosses. Its own pathology is the dying ReLU — a unit pushed permanently negative has zero gradient forever and can never recover — which leaky ReLU and its relatives patch by giving the negative side a small slope.

Initialisation is not a detail

Initialise all the weights to zero and every neuron in a layer computes the same thing, receives the same gradient, and stays identical forever: the network never breaks symmetry and behaves like a single unit per layer. So the weights must be random. But their scale matters just as much, because the variance of the signal is multiplied layer by layer — too large and activations explode, too small and they collapse to zero. The two standard schemes keep that variance at 1:

Xavier / Glorot   Var(W) = 2 / (fan_in + fan_out)   for tanh, sigmoid
He / Kaiming      Var(W) = 2 / fan_in                for ReLU
                  (the factor 2 compensates for ReLU zeroing half
                   the activations, which halves the variance)

From gradients to learning

With gradients in hand, the update rule is the last choice. Plain stochastic gradient descent takes a step against the gradient of a mini-batch. Momentum accumulates an exponentially decayed average of past gradients, which damps the oscillation across the steep walls of a ravine and accelerates along its floor. Adam keeps running estimates of both the first and the second moment of the gradient and divides by the square root of the second — giving every parameter its own effective learning rate.

SGD          w -= lr * g
Momentum     v = μ*v + g            ;  w -= lr * v          μ ≈ 0.9
Adam         m = β₁*m + (1−β₁)*g                             β₁ ≈ 0.9
             v = β₂*v + (1−β₂)*g²                            β₂ ≈ 0.999
             (bias-correct m̂, v̂ by 1−βᵗ)
             w -= lr * m̂ / (sqrt(v̂) + ε)                     ε ≈ 1e-8

The learning rate remains the single most consequential hyper-parameter: too large and the loss diverges or oscillates, too small and training stalls in a plateau. And when a network refuses to learn at all, the first thing to check is not the architecture but the gradients themselves — compare a few analytic gradients against a central finite difference, (L(w+ε) − L(w−ε)) / 2ε, on a tiny network with double precision. A backpropagation implementation with a sign error or a missing transpose looks exactly like a badly tuned one, and only gradient checking tells the two apart.

Frequently asked questions

Try it live

Everything above runs in your browser — open Neural Network — Backpropagation and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.