LoRA Adapter Lab: Low-Rank Fine-Tuning

Task Adapter

Adapter Shape

Rank r 4

LoRA α (scale numerator) 8.0

Learning rate 0.35

Live Readouts

Task loss (MSE)

0.000

Train steps

Trainable params

Full fine-tune params

Parameter savings vs. full fine-tune

How it works

Fine-tuning a whole weight matrix W (size d×d) means training d² numbers per task. LoRA instead freezes the pretrained base W₀ and learns a low-rank update:

W_eff = W0 + (α / r) · B·A
B: d×r,  A: r×d,  rank r ≪ d
B initialised to 0, so training starts
with ΔW = 0 — the frozen base is
untouched until the adapter learns.

Only B and A are trained per task — that's 2·d·r numbers instead of d². For d=14 and r=4 that's 112 trainable numbers versus 196 for a full fine-tune, and the gap widens fast as d grows in real transformers (d in the thousands, r as low as 4–16).

Each task here has its own target pattern the adapter tries to reach by gradient descent on the mean-squared error between W_eff and the target:

L = mean((W_eff − Target)²)
∂L/∂B = (α/r) · (W_eff − Target) · Aᵀ
∂L/∂A = (α/r) · Bᵀ · (W_eff − Target)

Task A / Task B buttons — swap in a different adapter and target; the base grid W0 never changes, so switching tasks can never overwrite what a previous adapter learned. That's how a real LoRA-fine-tuned model avoids catastrophic forgetting: task-specific knowledge lives entirely in a small, separately-stored adapter.
Train — runs real gradient-descent steps on B and A only.
Rank r — raises the adapter's capacity to match the target pattern; the parameter-savings readout updates live.
α (alpha) — LoRA's scaling numerator; the applied scale is always α/r, the standard convention from the LoRA paper (Hu et al., 2021), which keeps the update's magnitude roughly rank-independent.

Real-world relevance: this is exactly how adapters and LoRA modules let large language models pick up new skills or domains without the cost — or the forgetting risk — of updating every one of their billions of frozen weights.

W_eff = W0 + (α/r)·B·A (center) B (d×r, left) A (r×d, right)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.

LoRA Adapter Lab: Low-Rank Fine-Tuning Without Forgetting