Dying ReLU: Watching Neurons Go Dark

Training

Learning rate 0.30

Initial weight scale 1.0

Leaky ReLU slope α 0.00

Live Readouts

Dead neurons

0 / 24

Epoch

Training loss (MSE)

—

How it works

Each of the 24 hidden neurons computes a pre-activation from a scalar input x, then a ReLU:

z_i   = w_i·x + b_i
h_i   = ReLU(z_i) = max(0, z_i)
y_hat = Σ v_i·h_i + c

Gradient descent updates every weight by the partial derivative of the mean-squared error, scaled by the learning rate. The derivative of ReLU is the step function:

dReLU/dz = 1  if z > 0
         = 0  if z ≤ 0   (plain ReLU)
         = α  if z ≤ 0   (Leaky ReLU)

If a large update ever pushes z_i ≤ 0 for every training point at once, that neuron's gradient becomes exactly zero under plain ReLU (α = 0). No future step can move w_i or b_i again — the neuron is permanently dead, forever outputting 0 regardless of input. Aggressive learning rates and large initial weights make this overshoot far more likely, especially early in training.

Learning rate — bigger steps fit faster but overshoot more, killing more neurons.
Initial weight scale — larger starting weights push more neurons into the dead zone from the very first step.
Leaky α — a small non-zero slope for z ≤ 0 keeps a trickle of gradient alive, so a neuron that looked dead can still recover.

This is the exact mechanism behind the real "dying ReLU" problem in deep learning, and the reason Leaky ReLU, PReLU, ELU and GELU exist as drop-in replacements.

Target function Network prediction Active neuron Dead neuron

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.