LoRA: Low-Rank Adapters for Fine-Tuning

Low-Rank Adaptation (LoRA) is the technique that makes it practical to fine-tune huge language models on modest hardware: instead of updating every weight in a d×k matrix, it freezes the original weights and learns a small correction factored as ΔW = (α/r)·B·A, where B and A are skinny d×r and r×k matrices. This simulator runs real gradient descent, live, training B and A to approximate a fixed rank-8 target correction, and renders both matrices as 3D bar grids so you can watch the learned update converge toward the target cell by cell. Sweep the rank slider below and above the target's true rank to see the reconstruction-loss plateau appear and disappear, and watch the trainable-parameter and memory-savings readouts show exactly why a rank-4 or rank-8 adapter needs a small fraction of a full fine-tune's parameters.