LoRA: Low-Rank Adapters for Fine-Tuning

Adapter Configuration

Rank r 4

Scaling α 8

Learning rate η 0.40

Live Readouts

Training step

Reconstruction loss

—

Trainable params

32 / 256

Memory vs full fine-tune

12.5%

How it works

Full fine-tuning of a d×k weight matrix W means learning d·k new numbers. LoRA instead freezes W and learns a much smaller update ΔW, factored into two skinny matrices:

ΔW = (α / r) · B · A
B is d×r,  A is r×k,  r << min(d,k)
W_new = W_frozen + ΔW

trainable params = r·(d + k)   instead of   d·k

Here d = k = 16, so a full update would need 256 parameters. This panel trains B and A by gradient descent on a fixed, hidden target update ΔW* that was built with rank 8:

L = ‖ΔW* − (α/r)·B·A‖²_F  (mean squared error over all 256 cells)
∂L/∂B = −2(α/r)·(ΔW* − ΔW)·Aᵀ
∂L/∂A = −2(α/r)·Bᵀ·(ΔW* − ΔW)
B ← B − η·∂L/∂B,  A ← A − η·∂L/∂A

Rank r — how many rank-1 "layers" the adapter has. Below the target's true rank (8) the loss plateaus above zero — the adapter is not expressive enough. At or above rank 8 it can, in principle, reach zero loss.
Scaling α — the update actually applied is (α/r)·B·A, the standard LoRA convention; raising α without retraining scales the correction up.
Learning rate η — step size of the gradient descent shown live on the canvas.
Trainable params / memory — r·(d+k) vs the 256 a full fine-tune would need — this ratio is why LoRA lets huge LLMs be adapted with a tiny checkpoint.

Real-world relevance: this is the exact mechanism behind LoRA and QLoRA adapters used to fine-tune large language models cheaply — instead of storing a full copy of every weight matrix per task, only the small A/B factors are saved and swapped in at inference time.

loss — rank 4

Target ΔW* (frozen, rank 8) Learned ΔW = (α/r)·B·A Factors A (top) / B (right)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.