PEFT / LoRA Rank Bottleneck 3D

LoRA Rank Bottleneck

Rank r 8

Matrix dimension d 4096

LoRA scale α 16

Frozen params (W)

–

Trainable params (A+B)

–

Trainable share

–

Compression d÷2r

–

How it works

The wide, dimmed platform of cubes is the frozen weight matrix W — every one of its d² entries stays fixed during LoRA training. Instead of updating that whole grid, a forward pass also runs through a narrow funnel: a wide plate compresses down to just r channels (the rank bottleneck), then a second plate expands back out to d. That funnel is exactly A and B — down-projection then up-projection — and its waist width is the rank r.

h = W·x + ΔW·x
ΔW = (α/r) · B·A
waist width = r  ≪  plate width = d

Rank r — the waist of the funnel; drag it up and the bottleneck widens, more trainable channels, more expressive ΔW.
Matrix dimension d — how wide the frozen platform (and the funnel's plates) are; the ratio d:r is the whole point of LoRA's efficiency.
LoRA scale α — brightens/dims the pulses travelling through the funnel, reflecting how strongly ΔW's output is weighted before being added back to W's output.
Data pulses — toggle the glowing pulses that travel from the wide input plate, squeeze through the r-wide waist, and expand back out — a spatial view of the same bottleneck the 2D matrix-multiply view shows numerically.

Unlike the 2D grid-multiplication view, this scene never actually multiplies matrices — it's a geometric metaphor: the funnel's waist radius is driven directly by r/d, so the bottleneck you see is literally proportional to the compression ratio.

Frozen W A plate (in) B plate (out) Data pulse

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.

PEFT / LoRA Rank Bottleneck — 3D