Weight Initialization & Vanishing Activations Lab

Weight Initialization

Network

Hidden layers 4

Activation

Learning rate 0.50

Training

Epoch

Loss (MSE)

0.250

Accuracy

50%

Deep-layer act. σ

0.00

How it works

A feedforward network of L layers is trained on a 2D circle-classification task by real backpropagation, running fully in your browser. Each hidden layer l computes

z_l = W_l · a_(l-1) + b_l
a_l = φ(z_l)

and gradients flow back via the chain rule, δ_l = (W_(l+1)ᵀ δ_(l+1)) ⊙ φ'(z_l). The 3D view renders every layer as a slab of bars — bar height and colour encode the standard deviation of that neuron's activation across the training batch.

Weight initialization sets the scale of W_l before training starts, and that scale compounds across layers:

Zeros:   W = 0                → every neuron identical, no gradient signal at all
Large:   W ~ N(0, 1)          → variance grows layer to layer, saturates φ, gradients vanish
Xavier:  W ~ N(0, 1/fan_in)   → keeps variance ≈1 through sigmoid/tanh (Glorot & Bengio, 2010)
He:      W ~ N(0, 2/fan_in)   → compensates ReLU zeroing half its inputs (He et al., 2015)

Zeros — every neuron in a layer computes the same value and gets the same gradient forever; the network can never break symmetry.
Large random — with sigmoid/tanh, pre-activations grow with depth and saturate to ±1, where φ'≈0: the deep-layer σ readout collapses toward 0 (vanishing activations/gradients) even though the weights themselves are large.
Xavier / He — chosen so the forward-pass variance stays roughly constant across depth, which is why deep networks became trainable at all; He additionally accounts for ReLU discarding negative inputs.
Switch to ReLU with Xavier init to see it still drift (Xavier assumes a symmetric activation); switch to He to see it stabilise.

Low σ (vanishing) Healthy σ High σ (exploding)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.