Vanishing Gradients in RNN Backpropagation

Recurrent Network

Sequence length 18 steps

Recurrent weight w 0.90

Activation function

Playback speed 1.0×

Backward Pass

—

Current step t

–

|dL/dh_t|

–

|dL/dh_1| (earliest)

–

Gradient horizon

–

How it works

An RNN updates a hidden state every step: h_t = f(w·h_t−1 + u·x_t). Training needs ∂L/∂h_t for every t, found by backpropagation through time — the chain rule applied once per unrolled step:

∂L/∂h_t = ∂L/∂h_T · ∏_{k=t+1..T} [ w · f'(pre_k) ]

local Jacobian per step:  J_k = w · f'(pre_k)
tanh:  f'(z) = 1 − tanh(z)²   (max value 1, at z=0)
ReLU:  f'(z) = 1 if z>0 else 0

Each backward step multiplies the running gradient by another local Jacobian J_k. If |J_k| is consistently below 1 (small |w|, or tanh saturated near ±1 where f'≈0), the product shrinks geometrically — the classic vanishing gradient: after a few dozen steps the earliest time steps get essentially zero learning signal. If |J_k| stays above 1, the product grows geometrically instead — the exploding gradient — which is why RNN training clips gradients. This scalar model captures the same exponential mechanism (Hochreiter 1991; Bengio et al. 1994) that motivated LSTM/GRU gating: a gated cell state gives the backward path an additive shortcut with local Jacobian ≈ 1, instead of a repeated multiplication by w·f'.

Recurrent weight w — the shared weight multiplied in at every step; |w| < 1 biases toward vanishing, |w| > 1 toward exploding.
Activation — tanh caps f' at 1 (always shrinks or holds), ReLU's f' is exactly 0 or 1 (kills or fully passes the gradient at each step).
Run backward pass — animates ∂L/∂h_t propagating from the last time step (seeded at 1) back to the first, one multiplication at a time.
Gradient horizon — how many steps back the gradient survives above 10⁻³ before it's numerically indistinguishable from zero.

Forward hidden state |h_t| Backward gradient |dL/dh_t| Current backward step

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.

Vanishing Gradients in RNN Backpropagation Through Time