LoRA 2D: Low-Rank Adapters for Fine-Tuning

Adapter Configuration

Rank r 4

Scaling α 8

Learning rate η 0.40

Training speed 20/s

Live Readouts

Training step

Reconstruction loss

—

Trainable params

32 / 256

Memory vs full fine-tune

12.5%

How it works

Full fine-tuning of a d×k weight matrix W means learning d·k new numbers. LoRA instead freezes W and learns a much smaller update ΔW, factored into two skinny matrices:

ΔW = (α / r) · B · A
B is d×r,  A is r×k,  r << min(d,k)
W_new = W_frozen + ΔW

trainable params = r·(d + k)   instead of   d·k

Here d = k = 16, so a full update would need 256 parameters. This panel trains B and A by gradient descent on a fixed, hidden target update ΔW* that was built with rank 8:

L = ‖ΔW* − (α/r)·B·A‖²_F  (mean squared error over all 256 cells)
∂L/∂B = −2(α/r)·(ΔW* − ΔW)·Aᵀ
∂L/∂A = −2(α/r)·Bᵀ·(ΔW* − ΔW)
B ← B − η·∂L/∂B,  A ← A − η·∂L/∂A

Rank r — how many rank-1 "layers" the adapter has. Below the target's true rank (8) the loss plateaus above zero — the adapter is not expressive enough. At or above rank 8 it can, in principle, reach zero loss.
Scaling α — the update actually applied is (α/r)·B·A, the standard LoRA convention; raising α without retraining scales the correction up.
Learning rate η — step size of the gradient descent, also shown as the slope of the loss curve.
Training speed — how many gradient steps are taken per second of wall-clock time (visualization pacing only, does not change the math).
Trainable params / memory — r·(d+k) vs the 256 a full fine-tune would need — this ratio is why LoRA lets huge LLMs be adapted with a tiny checkpoint.

Real-world relevance: this is the exact mechanism behind LoRA and QLoRA adapters used to fine-tune large language models cheaply — instead of storing a full copy of every weight matrix per task, only the small A/B factors are saved and swapped in at inference time.

2D vs 3D: the companion 3D simulator renders the same matrices as instanced bar grids you orbit around; this 2D version renders them as flat heatmaps plus a live scrolling loss curve, so you can watch convergence speed and the rank-8 plateau directly as a graph. Hover any cell to read its exact value.

loss — rank 4

Target ΔW* (frozen, rank 8) Learned ΔW = (α/r)·B·A Factors A (top) / B (right)