Presets
Playback
Model and memory
Cache
Camera
Layers
Quality
Model check
The science
What a transformer repeats when it writes text
A decoder writes one token at a time. To choose token t each attention layer compares a query q_t with the keys k_i of all earlier tokens and mixes their values v_i:
a_i = softmax_i( q_t . k_i / sqrt(d_h) ), out_t = sum_i a_i v_i, i = 1..t (per head, per layer) k_i = W_K x_i, v_i = W_V x_i
Because the model is causal, x_i for an old token never changes, so its k_i and v_i never change either. Without a cache the decoder re-projects them for the whole prefix at every step; with a cache it stores them once and reads them back.
Counting the work and the memory
K/V projections (per layer): with cache n, without cache n(n+1)/2 -> ratio (n+1)/2 attention score products: with cache n(n+1)/2, without n(n+1)(n+2)/6 -> ratio (n+2)/3 cache memory = 2 * L * n * d * b (K and V, L layers, width d, b bytes per number)
For a 7B-class model (32 layers, d = 4096, fp16) that is 2 x 32 x 4096 x 2 = 0.5 MiB per token, 2 GiB for a 4096-token context. The memory slider and the tank in 3D use this formula; the little decoder itself has d = 32 so it can be run in your browser.
What to look for
- Switch the cache off: each step now touches the whole prefix, the red curve is a parabola and the grid glows red where the same K/V pair has been re-projected many times.
- With the cache on the work per step is constant for projections, but the attention bars show the newest token still reads every cached key, so the cost of attention itself keeps growing with context. The cache removes redundant projections, not attention.
- Memory is the price: raise the layers or the width, or shrink the budget, and the tank fills before the context does.
Honest simplifications
The demo decoder is tiny and random: 4 heads, d = 32, two-layer-style blocks without layer norm, a 48-token vocabulary, no training, so the generated tokens are meaningless. Real systems add grouped-query attention (smaller caches), paged caches, quantised caches and batching, and measure wall-clock time, which depends on hardware. The counts here are operation counts, not timings. The cache and the recompute path give the same hidden state because they perform the same arithmetic in the same order, so a difference of exactly zero is expected; the check proves the cached bookkeeping is right, not that floating-point order never matters.