Home▸Machine Learning & Neural Networks▸Dying ReLU: Watching Neurons Go Dark

Dying ReLU: Watching Neurons Go Dark

Train a small neural network live in 3D and watch individual ReLU neurons go permanently dark: raise the learning rate or the initial weight scale and see a fraction of the hidden layer die, then bring them back with a Leaky ReLU slope.

Machine Learning & Neural Networks3DModerate60 FPS📱 Mobile-adapted⇄ 2D version

A 24-neuron hidden layer trains live, in your browser, to fit a wiggly target curve using ordinary gradient descent. Every neuron is drawn as a 3D bar that rises and falls with its own activation, turning red the instant its pre-activation drops below zero for every training example at once — the exact moment a plain ReLU neuron's gradient becomes zero and it can never update again. Push the learning rate or the initial weight scale up and watch a chunk of the layer die within seconds; dial in a Leaky ReLU slope and see previously "dead" bars flicker back to life as a small negative-side gradient starts flowing through them again.

⚙ Under the hood

Train a 24-neuron hidden layer live in 3D and watch individual ReLU neurons turn permanently dark: push the learning rate or initial weight scale up and see a chunk of the layer die, then dial in a Leaky ReLU slope to bring them back.

deep learningneural networksrelugradient descentbackpropagationmachine learning

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install