Residual Connections: Why Deep Networks Need Shortcuts

The Degradation Problem

Common sense suggests that stacking more layers onto a neural network should never make it worse, since a deeper network can always mimic a shallower one by having its extra layers learn the identity function. But in practice, researchers found the opposite: as plain networks got deeper, training accuracy would improve, then plateau, then get worse — even on the training set itself, not just unseen data. This ruled out overfitting as the culprit and pointed to something more fundamental: plain deep networks were simply hard to optimize, because gradients and signals struggled to propagate cleanly through dozens of stacked layers.

Learning the Residual, Not the Whole Mapping

The fix proposed by He et al. was to change what each block of layers is asked to learn. Instead of forcing a stack of layers to directly learn a full desired mapping H(x), residual blocks let them learn only the residual F(x) = H(x) − x, while a skip connection carries the original input x forward and adds it back in. The output becomes F(x) + x, so if the ideal transformation for a block is close to doing nothing, the layers just need to push F(x) toward zero — a far easier target than reconstructing the identity mapping from scratch through nonlinear layers.

Why Shortcuts Make Optimization Easier

The real payoff of residual connections shows up during backpropagation. Because the skip connection provides a direct, unimpeded path from output back to input, gradients can flow through it without being repeatedly multiplied by small weight-layer derivatives at every step — a major source of the vanishing gradient problem in very deep plain networks. This means even the earliest layers of a 100+ layer network keep receiving strong, meaningful gradient signals, so the whole stack can be trained effectively with ordinary gradient descent rather than getting stuck or diverging.

From ImageNet to Everywhere

ResNet's impact was immediate and dramatic: it won the 2015 ImageNet Large Scale Visual Recognition Challenge with networks reaching 152 layers, far deeper than anything trained successfully before, and reduced top-5 error to a level that beat human-level benchmarks on that task. But the idea outgrew image classification entirely — residual connections are now a default architectural ingredient in transformers (including every layer of models behind modern language and vision AI), speech models, and reinforcement learning networks, wherever engineers need to train very deep systems reliably.

Frequently asked questions

Try it live

Everything above runs in your browser — open Residual Connections: Why Deep Networks Need Shortcuts and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.