Presets
Playback
Data
The 2D classification task.
Network
Hidden layers, neurons and activation function.
Training
Optimizer, learning rate and regularisation.
Layers
Model checks
The science
The network
A fully connected network maps an input x = (x1, x2) through layers. Layer l computes a pre-activation and an activation:
z(l) = W(l) a(l−1) + b(l) a(l) = f(z(l))The output neuron uses the logistic function σ(z) = 1/(1 + e−z) and is read as the probability that the point belongs to class 1. The coloured disks in the diagram are the activations a of every neuron for the probe point you click on the map; line thickness and colour show the weights.
Why a nonlinearity
Without f the whole network collapses to one linear map, so it could only draw a straight line through the plane. Hidden nonlinear units bend the boundary: XOR needs at least two hidden units, spirals need many more.
Learning by backpropagation
With binary cross-entropy loss L = −[y ln p + (1 − y) ln(1 − p)] + (λ/2)∑W², the output error is δ(L) = p − y, and the chain rule gives
δ(l) = (W(l+1))T δ(l+1) ⊙ f′(z(l)) ∂L/∂W(l) = δ(l) (a(l−1))TWeights are updated with SGD, momentum or Adam on mini-batches. The page checks the analytic gradient against a finite-difference estimate.
Vanishing gradients
Each backward step multiplies by f′. For the logistic function f′ ≤ 0.25, for tanh f′ ≤ 1, for ReLU f′ is 0 or 1. In a deep sigmoid network the gradient of the first layer can be orders of magnitude smaller than that of the last: watch the per-layer gradient graph while you add layers.
Honest limits
This is a tiny teaching network on synthetic 2D data with a training set and no separate validation set, so the accuracy shown is training accuracy and says nothing about generalisation. Real networks have far more inputs, normalisation layers, dropout and carefully tuned learning-rate schedules.
What to try
- Pick XOR with 1 hidden layer of 2 neurons and train: the boundary appears. With tanh and 1 neuron it can never succeed.
- Pick Spiral and increase layers and neurons until the training accuracy passes 95 %.
- Choose sigmoid with 4 hidden layers and compare the per-layer gradient graph with ReLU.
- Click the map to move the probe and watch the neuron activations change.
- Raise the learning rate until the loss curve turns unstable.