Neural Network Activation Explorer

Train a small neural network on 2D data and see every neuron's activation, the decision boundary, the loss and the gradient in each layer.

Presets

Playback

Data

The 2D classification task.

Network

Hidden layers, neurons and activation function.

Training

Optimizer, learning rate and regularisation.

Layers

Model checks

The science

The network

A fully connected network maps an input x = (x1, x2) through layers. Layer l computes a pre-activation and an activation:

z(l) = W(l) a(l−1) + b(l)    a(l) = f(z(l))

The output neuron uses the logistic function σ(z) = 1/(1 + e−z) and is read as the probability that the point belongs to class 1. The coloured disks in the diagram are the activations a of every neuron for the probe point you click on the map; line thickness and colour show the weights.

Why a nonlinearity

Without f the whole network collapses to one linear map, so it could only draw a straight line through the plane. Hidden nonlinear units bend the boundary: XOR needs at least two hidden units, spirals need many more.

Learning by backpropagation

With binary cross-entropy loss L = −[y ln p + (1 − y) ln(1 − p)] + (λ/2)∑W², the output error is δ(L) = p − y, and the chain rule gives

δ(l) = (W(l+1))T δ(l+1) ⊙ f′(z(l))    ∂L/∂W(l) = δ(l) (a(l−1))T

Weights are updated with SGD, momentum or Adam on mini-batches. The page checks the analytic gradient against a finite-difference estimate.

Vanishing gradients

Each backward step multiplies by f′. For the logistic function f′ ≤ 0.25, for tanh f′ ≤ 1, for ReLU f′ is 0 or 1. In a deep sigmoid network the gradient of the first layer can be orders of magnitude smaller than that of the last: watch the per-layer gradient graph while you add layers.

Honest limits

This is a tiny teaching network on synthetic 2D data with a training set and no separate validation set, so the accuracy shown is training accuracy and says nothing about generalisation. Real networks have far more inputs, normalisation layers, dropout and carefully tuned learning-rate schedules.

What to try
  • Pick XOR with 1 hidden layer of 2 neurons and train: the boundary appears. With tanh and 1 neuron it can never succeed.
  • Pick Spiral and increase layers and neurons until the training accuracy passes 95 %.
  • Choose sigmoid with 4 hidden layers and compare the per-layer gradient graph with ReLU.
  • Click the map to move the probe and watch the neuron activations change.
  • Raise the learning rate until the loss curve turns unstable.
Training loss and accuracy
Activation function and its derivative
Mean absolute gradient per layer (vanishing gradients)
Weight histogram