Gradient Clipping 2D: Exploding & Vanishing Gradients

Network Depth

Layers L 24

Per-Layer Gradient Factor

Mean |w·σ'| per layer 1.15

Gradient Clipping

Clip norm threshold 2.00

Live Readouts

Raw gradient norm ‖g‖ at layer 1 (input)

—

Norm after clipping

—

Clip scale

1.000

Regime

stable

How it works

Backpropagating through a chain of L layers multiplies the gradient by a per-layer factor at every step (chain rule). For the gradient of the loss with respect to the earliest hidden state:

dL/dh₀ = ∏(i=1..L) wᵢ · σ'(zᵢ)

Each factor wᵢ·σ'(zᵢ) here is drawn around the "Mean factor" slider with random per-layer jitter. If the mean magnitude is > 1, the product grows exponentially with depth — exploding gradients. If it is < 1, the product shrinks toward zero — vanishing gradients. Both make deep networks (and long RNN unrolls) hard to train: exploding gradients cause huge, destabilizing weight updates; vanishing gradients mean early layers barely learn at all.

Global-norm gradient clipping (used in virtually every modern deep-learning framework) rescales the whole gradient vector once its norm exceeds a threshold, preserving direction but capping magnitude:

scale = min(1, threshold / ‖g‖)
g ← g · scale

Layers L — depth of the backprop chain; longer chains amplify the same per-layer effect.
Mean factor — average multiplicative gain per layer; above 1.0 the chain trends toward exploding, below 1.0 toward vanishing.
Clip norm threshold — the cap applied to the gradient norm when clipping is on.
Clipping toggle — compare the raw exponential blow-up/decay against the clipped, stabilized version.

The top row of nodes shows each layer's gradient magnitude on a log color/size scale, with a pulse animating the actual backward flow of information from the output (right) toward the input (left) — exactly the direction backpropagation travels. The curve below plots log₁₀(magnitude) against layer index, so the exponential blow-up or decay reads as a straight line whose slope is set by the mean factor. Drag to pan and scroll to zoom the view.

drag to pan · scroll to zoom

← backprop direction (output → input)

~1.0 (stable) growing (exploding) shrinking (vanishing)