Dying ReLU in Function Space

A 24-neuron hidden layer trains live, in your browser, to fit a wiggly target curve using ordinary gradient descent — the same network as the 3D bar version, seen through a genuinely different lens. Instead of a bar rising and falling per neuron, every hidden unit is drawn as its actual piecewise-linear hinge function v_i·ReLU(w_i·x + b_i), overlaid directly on the x–y plane the network is fitting. A small triangle marks each hinge's breakpoint on the x-axis — the point where the neuron switches from flat to sloped. Push the learning rate or initial weight scale up and watch a cluster of hinges snap flat and turn red, their breakpoints frozen in place, the instant their gradient goes to exactly zero; dial in a Leaky ReLU slope and see previously frozen hinges start tilting again as a small negative-side gradient starts flowing through them.