Controls
Presets
Training
Dataset
Hidden activation
Optimiser
Layers
Experiments
Model checks
The science
The network. A multilayer perceptron maps a point (x, y) to the probability p that it belongs to class 1. Layer l computes a_l = f(W_l a_(l-1) + b_l), with f = tanh, ReLU or sigmoid in the hidden layers and a sigmoid at the output, p = sigma(z). The weights and biases on screen are the real ones being trained, not an illustration.
The loss. Binary cross-entropy plus an optional L2 term: L = -(1/N) sum [ y ln p + (1-y) ln(1-p) ] + (lambda/2) sum W^2. Predicting p = 1/2 everywhere gives L = ln 2 = 0.693, the level every run starts near.
Backpropagation. With a sigmoid output and cross-entropy the output error is simply delta_L = p - y. It is passed backwards with the chain rule, delta_l = (W_(l+1)^T delta_(l+1)) * f'(z_l), and the gradients are dL/dW_l = (1/N) sum delta_l a_(l-1)^T + lambda W_l and dL/db_l = mean(delta_l). The pink rings show the mean size of delta per neuron, the pink pulses the size of dL/dw per weight (relative to the largest one).
Optimisers. SGD: w = w - eta g. Momentum: v = 0.9 v + g, w = w - eta v. Adam: running averages of g and g^2 with beta1 = 0.9, beta2 = 0.999, bias correction and eps = 1e-8. Weights start from a seeded Glorot (He for ReLU) draw, so Reset repeats a run exactly and New seed gives another start.
The data. Spiral: two interleaved arms, r = 0.95 t, angle = 2 pi (1.15 t) + c pi, plus Gaussian noise; XOR, circles and moons are the usual toy sets. The test set is 200 fresh points from the same generator with another seed. A gap between the train and test curves is overfitting; L2 shrinks it only a little on this tiny set.
Simplifications. Full-batch gradient descent (no mini-batch noise), no dropout or batch normalisation, at most 151 parameters, double precision JavaScript. The 3D layout, the pulse motion and the glow are visual aids: pulse positions do not represent signal timing. This is a classroom model of supervised learning, not a claim about how brains learn.
Checks on this page. (1) backpropagation against central finite differences, (2) the loss of an untrained output layer against ln 2, (3) XOR, which no single line can solve, learned by a 2-4-1 net, (4) the parameter count. The "Gradient check now" button repeats (1) on the weights you are training right now. Both the 2D and the 3D page import the same model file, so the same settings give identical numbers.
What to try
- Spiral, Adam: the boundary unwinds into two arms within a few hundred epochs.
- Plain SGD plateau: the loss sits near 0.65 for a long time, then drops; Momentum and Adam skip most of that wait.
- Too small 2-3-3-1: the network cannot bend enough and stays at about 90 %.
- Overfitting: train accuracy reaches 100 % while test accuracy stays near 65 %; add L2 and see the small difference.
- Sigmoid, SGD: very small gradients in the first layer (third graph) and almost no learning, the vanishing-gradient problem.
- Learning rate too high: Adam with eta = 2 overshoots and never settles.
- Learning-rate sweep: epochs to reach 95 % train accuracy for six learning rates.