The grid in front of you is a toy "network" of 144 weights. Every task pulls those weights toward its own target pattern. When a new task trains freely, it overwrites weights the old task depended on — catastrophic forgetting. Elastic Weight Consolidation (EWC) fights this by penalising change to weights that mattered for earlier tasks:
L(θ) = L_new(θ) + (λ/2) · Σ_i F_i · (θ_i − θ*_i)²
θ_i current value of weight i
θ*_i value weight i had right after the previous task
F_i Fisher information ≈ how much task performance
depended on weight i (accumulated every task switch)
λ EWC strength — the slider on the left
effective step: θ_i ← θ_i + lr · grad_i / (1 + λ·F_i)
- Next task → — starts training on a brand-new target pattern that partly overlaps the previous ones (just like real tasks sharing input features/weights).
- EWC strength λ — at 0 you get plain fine-tuning: the bars for old tasks collapse the moment a new one starts. Raise it and important weights (amber) barely move, so old tasks stay accurate at the cost of slower learning on the new one — the classic plasticity vs. stability trade-off.
- Replay buffer — each step also nudges every weight back toward the average of all previously-learned targets, approximating interleaving stored samples from a memory buffer during training (experience replay).
- Forgetting — the average gap between a task's best-ever accuracy and its accuracy right now, averaged over every task except the one currently training.