Design architectures · Train with gradient descent · Watch decision boundaries evolve
The chain rule applied recursively: compute ∂L/∂w for every weight by propagating error gradients from output to input. Enables learning in networks with thousands of layers.
Non-linearities that allow networks to learn complex patterns. ReLU (max(0,x)) avoids vanishing gradients in deep nets; Sigmoid squashes to (0,1); Tanh centres around zero.
Cross-entropy for classification; MSE for regression. Gradient descent navigates the loss surface by following the negative gradient — the direction of steepest descent.
A network with one hidden layer and enough neurons can approximate any continuous function (Cybenko 1989). Depth helps efficiency — each layer learns higher-level abstractions.
| Function | Formula | Output Range | Gradient | Best For | Weakness |
|---|---|---|---|---|---|
| Sigmoid | 1/(1+e⁻ˣ) | (0, 1) | f(1−f) | Output layer (binary) | Vanishing gradient in deep nets |
| Tanh | (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) | (−1, 1) | 1−f² | Hidden layers (zero-centred) | Still saturates at extremes |
| ReLU | max(0, x) | [0, ∞) | 0 or 1 | Deep networks (default) | Dying ReLU (neurons stuck at 0) |
| Leaky ReLU | max(0.01x, x) | (−∞, ∞) | 0.01 or 1 | Deep networks | Slight inconsistency for negative |
| ELU | x if x≥0; α(eˣ−1) else | (−α, ∞) | Smooth | Negative push (self-normalising) | Computationally expensive |
| GELU | x·Φ(x) | (−0.17, ∞) | Soft gate | Transformers (BERT, GPT) | More complex to compute |
| Swish | x·σ(x) | (−0.28, ∞) | Smooth | Deep networks (EfficientNet) | Requires parameter search |
| Softmax | eˣⁱ/Σeˣʲ | (0, 1) summing to 1 | p(1−p) | Multi-class output | Only valid at output layer |
Backpropagation applies the chain rule to compute ∂L/∂w for every weight — propagating error gradients backwards from output to input. Each weight is updated: w ← w − α·(∂L/∂w). One forward + backward pass is one training step. The algorithm runs in O(|weights|) time, making it tractable for millions of parameters.
Without non-linear activations, any stack of linear layers collapses to a single linear map: W₂(W₁x) = (W₂W₁)x. No matter how many layers, only linear decision boundaries are possible. Non-linear activations (ReLU, sigmoid, tanh) allow the network to compose non-linear transformations — proven sufficient to approximate any function by the Universal Approximation Theorem.
Overfitting occurs when the model has too many parameters relative to training data. The network memorises training examples rather than learning general patterns, producing low training loss but high validation loss. Remedies: dropout (randomly zero activations), L2 regularisation (weight decay), early stopping, data augmentation, or reducing model size.
Cybenko (1989) and Hornik (1991) proved that a single hidden layer with enough neurons and a non-linear activation can approximate any continuous function on a compact domain to arbitrary precision. This gives theoretical justification for neural networks — though it does not say how many neurons are needed or whether gradient descent will find the solution.
CNNs, RNNs, transformers, attention mechanisms, training techniques, and regularisation — all explained.
Read Deep Learning Guide →