Backpropagation: The Chain Rule That Trains Neural Networks

One old rule, applied systematically

Backpropagation is the algorithm that makes training deep networks tractable, and at its core it is nothing more exotic than the chain rule from first-year calculus, applied to every parameter in a network at once. If f(x) = g(h(x)), the chain rule says df/dx = dg/dh · dh/dx — the derivative of a composition is the product of the derivatives of its parts. Rumelhart, Hinton and Williams popularised this as a training method in 1986, and the field has run on some version of it ever since: for a chain of n composed operations, the derivative is a product of n local partial derivatives, and backprop is exactly that product, evaluated layer by layer starting from the loss.

Turning an expression into a graph

Any mathematical expression can be represented as a computational graph — a directed acyclic graph whose nodes are operations (add, multiply, exponentiate, apply an activation) and whose edges carry values forward. Take a single squared-error loss L = (a·w + b − y)²: it decomposes into a multiply, an add, a subtract, and a square, chained in sequence. The forward pass evaluates this graph left to right, computing and caching every intermediate value. The backward pass then walks the same graph right to left, multiplying local derivatives to accumulate each node's contribution to the total gradient of the loss with respect to every input — that right-to-left multiplication is literally the chain rule, executed once per edge.

forward:   m = a*w        s = m+b        e = s-y        L = e^2

backward (seed dL/dL = 1):
  dL/de = 2e
  dL/ds = dL/de * de/ds = 2e
  dL/dm = dL/ds * ds/dm = 2e
  dL/dw = dL/dm * dm/dw = 2e * a
  dL/da = dL/dm * dm/da = 2e * w

When a value feeds into more than one downstream node — a weight shared across a layer, an input reused in several places — its gradient is the sum of the contributions flowing back from every node that used it. That's the multivariate chain rule showing up directly in the accounting, and it is the detail that trips up most from-scratch implementations if it's missed.

From scalars to matrices: the Jacobian

Real networks operate on vectors and matrices, not single numbers, so the "derivative" of a layer becomes a Jacobian matrix J where J_ij = ∂y_i/∂x_j. For a fully connected layer y = Wx + b, the useful gradients turn out to be an outer product and a matrix-vector product rather than anything requiring the full Jacobian to be materialised explicitly:

dL/dW = delta . x^T        (outer product; delta = dL/dy, upstream gradient)
dL/dx = W^T . delta         (passed further back to the previous layer)
dL/db = delta               (summed over the batch dimension)

Element-wise activation functions keep their Jacobian diagonal, collapsing to a simple pointwise product with the upstream gradient. Softmax combined with cross-entropy loss is the pleasant exception where a genuinely dense Jacobian simplifies algebraically all the way down to ŷ − y, which is why that combination is used almost universally for classification outputs rather than computed the long way.

Why gradients vanish — and how to stop it

A 20-layer network's earliest-layer gradient is a product of roughly 20 weight matrices and 20 activation derivatives, chained together by exactly the mechanism above. If each factor's magnitude sits below 1 — which happens with a saturating activation like sigmoid, whose derivative never exceeds 0.25 — that product shrinks geometrically: 0.25²⁰ is on the order of 10⁻¹², a vanishing gradient that leaves the earliest layers essentially unable to learn. Push the factors above 1 instead and the same multiplication explodes rather than vanishes. Modern architectures counter this on several fronts at once: ReLU activations whose derivative is exactly 1 for any positive input, avoiding saturation outright; residual connections that give the gradient a shortcut path around whole blocks of layers; batch normalisation, which keeps pre-activations in a well-behaved range; and initialisation schemes like He or Xavier that are chosen specifically so the variance of activations neither grows nor shrinks layer over layer.

A minimal autograd engine

The whole mechanism above — build a graph on the forward pass, walk it backward multiplying local derivatives — fits in a surprisingly small amount of code. Each value tracks its own gradient, its inputs, and a small closure that knows how to push its own gradient onto those inputs; calling backward() topologically sorts the graph and fires every closure in reverse order.

class Value {
  constructor(data, children = []) {
    this.data = data; this.grad = 0;
    this._prev = children; this._backward = () => {};
  }
  mul(other) {
    const out = new Value(this.data * other.data, [this, other]);
    out._backward = () => {
      this.grad  += other.data * out.grad;
      other.grad += this.data  * out.grad;
    };
    return out;
  }
  backward() {
    const topo = [], seen = new Set();
    (function build(v){ if(!seen.has(v)){ seen.add(v);
      v._prev.forEach(build); topo.push(v); } })(this);
    this.grad = 1;
    topo.reverse().forEach(v => v._backward());
  }
}

This is reverse-mode automatic differentiation, and it is exactly what PyTorch's autograd and JAX's grad transform do under the hood, just at vastly larger scale with tensors instead of scalars. Reverse mode is the right choice for neural networks specifically because a loss has one scalar output and millions of parameter inputs — one backward pass recovers every parameter's gradient simultaneously, whereas forward-mode differentiation would need one pass per input dimension and only pays off in the opposite situation, many outputs and few inputs.

How the simulation here uses it

The Backpropagation simulation on this site trains a small multilayer perceptron live in the canvas, showing the forward pass computing activations left to right and the backward pass propagating gradients right to left through the same connections, coloured by sign and magnitude. Because the network is small enough to render every weight and every gradient on screen, you can watch a vanishing gradient happen in real time by stacking extra layers or switching to a saturating activation, and watch it resolve the moment you switch to ReLU or add a residual connection — the same trade-offs described above, but visible rather than abstract.

Frequently asked questions

Try it live

Everything above runs in your browser — open Backpropagation and watch the forward and backward passes light up the network as it trains. Nothing is installed, nothing is uploaded, the whole model lives in one tab.