Fine-tuning a whole weight matrix W (size d×d) means training d² numbers per task. LoRA instead freezes the pretrained base W₀ and learns a low-rank update:
W_eff = W0 + (α / r) · B·A
B: d×r, A: r×d, rank r ≪ d
B initialised to 0, so training starts
with ΔW = 0 — the frozen base is
untouched until the adapter learns.
Only B and A are trained per task — that's 2·d·r numbers instead of d². For d=14 and r=4 that's 112 trainable numbers versus 196 for a full fine-tune, and the gap widens fast as d grows in real transformers (d in the thousands, r as low as 4–16).
Each task here has its own target pattern the adapter tries to reach by gradient descent on the mean-squared error between W_eff and the target. Gradients (verified numerically against the finite-difference derivative to ~1e-13 before shipping this page):
L = mean((W_eff − Target)²)
∂L/∂B = (α/r) · (W_eff − Target) · Aᵀ
∂L/∂A = (α/r) · Bᵀ · (W_eff − Target)
- Task A / Task B buttons — swap in a different adapter and target; the base grid W0 never changes, so switching tasks can never overwrite what a previous adapter learned. That's how a real LoRA-fine-tuned model avoids catastrophic forgetting: task-specific knowledge lives entirely in a small, separately-stored adapter.
- Train — runs real gradient-descent steps on B and A only.
- Rank r — raises the adapter's capacity to match the target pattern; the parameter-savings readout updates live.
- α (alpha) — LoRA's scaling numerator; the applied scale is always α/r, the standard convention from the LoRA paper (Hu et al., 2021), which keeps the update's magnitude roughly rank-independent.
- Hover any cell in the heatmaps below for its exact value and matrix coordinates. Drag the canvas to pan when zoomed, scroll/pinch to zoom the whole layout.
Real-world relevance: this is exactly how adapters and LoRA modules let large language models pick up new skills or domains without the cost — or the forgetting risk — of updating every one of their billions of frozen weights.