A progressive network (Rusu et al., 2016) never overwrites a weight learned for an old task. Each new task k gets its own frozen column of layers; only the newest column trains. Old columns are locked, so nothing can drift and nothing is forgotten — the mechanism is structural isolation, not regularization.
h_k^(l) = f( W_k^(l) h_k^(l-1) + Σ_{i<k} U_k^(l:i) h_i^(l-1) )
W_k are the new column's own trainable weights; U_k are lateral "adapter" connections that read (but never write) the frozen activations h_i of every earlier column. Total parameter cost grows roughly linearly with task count:
Params(k) = Σ_j own(j) + [lateral] · Σ_j Σ_{i<j} lateral(i→j)
This 2D build exposes two constants the 3D original hard-coded, so you can feel their effect directly: transfer strength β sets how much each prior frozen column speeds up the new column's convergence (rate k = k0·(1+β·priorCount)); forgetting rate γ sets how much of its accuracy the naive baseline loses on every old task each time it trains a new one (accuracy *= γ).
- Add Task — freezes the current column's accuracy for good and grows a new one, capturing the lateral-connection setting at creation time.
- Train Current Task — advances only the newest column's learning curve; with lateral connections on, more prior columns speed convergence (positive transfer).
- Lateral connections — toggles the adapter pathways U_k on/off, for both the diagram and the parameter/transfer math.
- Drag the top diagram to pan across the growing column chain; scroll/pinch to zoom.
- Baseline bar — a single shared-weight network trained the same way: each training step on the newest task also perturbs the shared weights the old tasks depended on, so their accuracy visibly decays — the catastrophic forgetting this architecture is designed to avoid.
Trade-off: zero forgetting is bought with linearly growing memory and compute — the honest cost side of the isolation-vs-interference spectrum in continual learning.