Catastrophic forgetting happens because gradient descent on a new task can freely move weights that used to encode an old task, with nothing to hold them in place.
w_i(t+1) = w_i(t) - lr*grad_new + protect * lambda*(w_i(0)-w_i(t))
- Task nodes — learned parameters/skills, each tied to a previously mastered task.
- Task overlap — how much the new task's input distribution shares structure with old tasks.
- New-task learning rate — how aggressively gradients from the new task overwrite existing weights.
- Replay / EWC protection — elastic-weight-consolidation-style penalty (or rehearsal buffer) that resists overwriting important old weights.
Production continual-learning systems combine replay buffers, regularization (EWC), and modular adapters so a model can pick up new domains without erasing old capabilities.