🔁 Backpropagation

Stochastic gradient descent (SGD) updates weights after each single training example — fast but noisy. Full-batch gradient descent computes the exact gradient over the entire dataset before updating — stable but slow for large datasets. Mini-batch gradient descent (the standard in deep learning) uses a random subset of B examples per update, averaging the gradient over the batch. Mini-batch balances noise (which helps escape local minima) with computational efficiency (matrix operations over batches parallelise well on GPUs). This simulation runs single-sample SGD to show individual weight updates clearly.

Sigmoid σ(z) = 1/(1 + e^(–z)) squashes outputs to (0, 1) and was the original choice; it suffers from vanishing gradients for large |z|. Tanh(z) = (e^z – e^(–z))/(e^z + e^(–z)) maps to (–1, 1) and has stronger gradients near zero than sigmoid, making it better for hidden layers. ReLU(z) = max(0, z) is the most popular modern choice: it has a constant gradient of 1 for z > 0 (no vanishing), is computationally cheap, and produces sparse activations. Leaky ReLU and GELU are common variants that fix the "dying ReLU" problem.

The learning rate η scales each gradient step: W ← W – η ∂L/∂W. Too large an η causes the loss to oscillate or diverge (overshooting the minimum); too small an η leads to extremely slow convergence. Typical values range from η = 10⁻⁴ to 10⁻¹. The learning rate slider in this simulation uses a log scale (10^x where x goes from –3 to 0). Modern optimisers like Adam adapt η per-parameter using moment estimates, effectively setting a good η automatically and often requiring much less tuning.

Edge brightness during the backward pass is proportional to |∂L/∂w|, the absolute value of the gradient for that weight. Brightly glowing edges are learning fast — their weights are being updated by a large step. Dark edges correspond to nearly zero gradient, meaning those weights are barely changing. In deep networks with sigmoid activations you can often see that early layers have dim edges (vanishing gradient) while later layers glow brightly — illustrating exactly why deep networks were hard to train before ReLU and batch normalisation.

The canvas background is colour-coded by the network's current class prediction at every point: red regions are predicted as class 0, blue as class 1, with saturation indicating confidence. After each weight update this background is re-rendered by running a forward pass at every pixel — which is computationally expensive for large networks, so it is sampled on a coarser grid and upscaled. As training progresses you can watch the decision boundary contort and sharpen to separate the two classes, sometimes getting stuck in a local minimum.

The five presets test different aspects of network expressiveness: "2 Gauss" (linearly separable blobs) can be solved even without hidden layers; "XOR" requires at least one hidden layer because the classes are not linearly separable; "Moons" and "Spirals" require the network to learn a curved, non-convex boundary; "3 Cluster" tests multi-class separation with a softmax output. Spirals are a classic benchmark — a fully connected net with only two hidden units of tanh can solve them, while a linear classifier cannot.

The gradient norm |∇| = √(Σ (∂L/∂w)²) is the Euclidean length of the entire gradient vector concatenated across all weights. A large gradient norm means the loss surface is steep and the network is far from a minimum; a very small norm suggests convergence or a plateau. Monitoring |∇| is useful for detecting exploding gradients (|∇| suddenly very large) and vanishing gradients (|∇| near zero early in training). Gradient clipping sets a maximum allowed |∇| to stabilise training of recurrent networks.

Yes — XOR is not linearly separable, meaning no single line can separate the four input combinations, so a perceptron (no hidden layers) fails completely. A network with at least one hidden neuron using a nonlinear activation can learn XOR exactly. Rumelhart et al.'s 1986 paper used XOR as the key demonstration that backprop enables hidden units to develop useful internal representations rather than just being intermediaries. With η = 0.1 and tanh activations, the network here typically solves XOR in a few hundred SGD steps.