Home▸Machine Learning & Neural Networks▸Vanishing Gradients vs. Residual Connections (2D)

Vanishing Gradients vs. Residual Connections (2D)

A 2D companion to the 3D ResNet residual-connections lab: real backpropagation through a deep multi-layer network, plain vs. residual, with per-layer gradient-norm readouts showing the plain network's gradient vanish exponentially while the residual network's stays healthy.

Machine Learning & Neural Networks2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Details

CategoryMachine Learning & Neural Networks

RenderingCanvas 2D

DifficultyAdvanced

Target FPS60

ControlsDepth slider (6-60 layers) sets the shared network depth for both columns; Weight scale slider (0.50-1.30x) rescales each layer's random weight matrix, controlling how severely the plain network's gradient vanishes; Speed slider controls the animated backward-sweep pulse; Run/Pause toggles the pulse animation; Reset weights redraws new random weights and reruns forward+backward on both networks.

resnetvanishing gradientresidual connectionsskip connectionsbackpropagationdeep learningneural networks

Related simulations

Decision Tree LiveCART decision tree with Gini impurity splitting; axis-aligned splits emerge level by level…

Reinforcement LearningQ-learning agent navigates an 8×8 grid maze. Heatmap shows max Q-values; white arrows show…

Self-Organising Map24×24 Kohonen SOM (576 neurons, 3D RGB input). Gaussian neighbourhood update as α and σ… ⊞ All 236 in Machine Learning & Neural Networks →

This is the 2D companion to the 3D residual-connections lab. Instead of an orbiting 3D layer stack, it runs a real forward and backward pass through two small multi-layer networks side by side — one plain, one with identity skip connections — using the same random weights and the same input. Watch the per-layer gradient-norm curve: the plain network's line collapses toward zero as it approaches the input layers (the vanishing-gradient problem that made very deep networks untrainable before 2015), while the residual network's line stays close to 1 all the way through, because its extra identity path guarantees the gradient a way back that never depends on the layer's weights shrinking it.

⚙ Under the hood

A 2D companion to the 3D ResNet residual-connections lab that runs real backpropagation instead of an animated 3D layer stack: two small multi-layer networks sharing the same random weights and input -- one plain, one with identity skip connections -- get a real forward and backward pass every rebuild, and a per-layer gradient-norm chart shows the plain network's signal decay exponentially toward the input layers while the residual network's stays close to 1, exactly the degradation problem the original ResNet paper diagnosed.

resnetvanishing gradientresidual connectionsskip connectionsbackpropagationdeep learningneural networks

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install