KV Cache in 2D: Linear vs Quadratic Inference Cost

Autoregressive Generation

Auto-play speed 2 tok/s

Model depth (layers) 6

Model width d_model 1024

ready — 5 tokens pre-seeded

Live Readouts

Tokens generated

Cache memory

0.06 MB

K/V projection cost so far (cache vs. recompute-every-step)

15 vs. 15 units

Compute saved by caching

How it works

A transformer decoder generates one token at a time. At step n, the new token's query must attend over the keys/values of every earlier token:

Attn(Q_n, K_1..n, V_1..n) = softmax(Q_n·Kᵀ / √d) · V
K_i = x_i·W_K,  V_i = x_i·W_V   (per layer)

Computing K_i and V_i requires a full forward pass through every layer for token i. Without a cache, a naive decoder redoes that projection for all n tokens at every single step, because nothing from the previous step is kept:

no cache:  work(n) = 1+2+3+...+L = L(L+1)/2   → O(L²)
KV cache:  work(n) = 1 per step, L total       → O(L)

The KV cache stores every token's K and V vectors, in every layer, the moment they're computed. Each new step then only projects the one new token and reuses the rest from memory — the attention softmax itself still scans all n cached vectors (that part is unavoidable and stays O(n) per step either way), but the expensive linear-projection work collapses from quadratic to linear in sequence length.

Generate next token — appends one token to the timeline; the attention-weight panel below it fans out lines to every cached key, weighted by softmax similarity.
KV Cache toggle — with it off, every earlier token's K/V bars flash red (recomputed from scratch) each step, and the cost curve panel visibly bends upward into a parabola.
Model depth and d_model — scale the memory and compute readouts, since every layer keeps its own K/V cache sized by the model's hidden width.
Precision toggle — fp16 vs fp32 halves/doubles the bytes each cached number costs, a direct memory lever real inference servers use (quantized KV cache).
Drag inside the timeline panel to pan once the sequence outgrows the visible width; double-click to snap back to the newest token.

Real-world relevance: this is exactly why serving a long-context LLM costs growing GPU memory per active conversation — the KV cache, not the model weights, is often the dominant memory user during inference.

Key vector (cached) Value vector (cached) Newest token / query Recompute flash (cache off)

drag to pan · dbl-click to recenter