PEFT / LoRA Adapter Simulator

LoRA Adapter

Rank r 8

Matrix dimension d 4096

LoRA scale α 16

Update speed 1.0×

Frozen params (W)

–

Trainable params (A+B)

–

Trainable share

–

Compression d÷2r

–

How it works

Full finetuning updates every weight in a d×d matrix W directly — for a large model that's billions of trainable numbers. LoRA instead freezes W entirely and learns a low-rank correction ΔW, factored as the product of two small matrices: a down-projection A (r×d) and an up-projection B (d×r), with r far smaller than d. Only A and B are trained; W never moves.

h = W·x + ΔW·x
ΔW = (α/r) · B·A
A: r×d   B: d×r   r ≪ d

Rank r — the width of the bottleneck A and B are forced through; lower r means fewer trainable numbers but less expressive ΔW.
Matrix dimension d — the size of the frozen weight matrix being adapted (e.g. a Q/K/V projection); this is what stays untouched during training.
LoRA scale α — rescales ΔW's contribution (α/r) so its magnitude stays sensible as r changes, without retraining a new learning rate each time.
Merged W+ΔW — toggle between the frozen base grid alone and the base plus the low-rank correction added in, which is what a forward pass actually computes.

The two small grids (A, B) on the right are a live, genuinely-computed low-rank matrix product — every frame recomputes ΔW = B·A from the current A/B values as they take small training-style steps, so the pattern in ΔW is really the outer product of two thin matrices, not a canned animation.

Frozen W A (r×d) B (d×r) ΔW = B·A