Grouped-Query Attention: Shrinking the KV Cache

Attention Configuration

Query heads (H) 32

KV head groups (G) 8

Sequence length cached 4096 tok

Model depth (layers) 32

Live Readouts

Heads per KV group4 : 1

KV cache size (this run)—

Memory saved vs full MHA—

How it works

In standard multi-head attention (MHA) every one of the H query heads keeps its own key and value projection, so the KV cache — the running memory of every past token's K and V vectors that autoregressive decoding must keep around — grows with H. Grouped-Query Attention (GQA) splits the H query heads into G groups and gives every head in a group the same shared K and V head, cutting the cache by a factor of H⁄G while every query head still attends independently:

KV cache bytes = 2 × L × S × G × d_head × b
  L = transformer layers, S = cached sequence length,
  G = number of KV head groups, d_head = per-head dimension,
  b = bytes per stored value (2 for fp16/bf16)
  factor 2 = one tensor for K, one for V

G = H  → MHA  (no sharing, largest cache, highest quality)
G = 1  → MQA  (all heads share one KV pair, smallest cache)
1
              
                Query heads (H) / KV groups (G) — choose any G that divides H; the diagram regroups the query heads (outer ring, cones) around the shared KV heads (inner ring, spheres) live.
                Preset buttons — jump straight to the three named regimes for the current H.
                Sequence length / model depth — the two other multipliers in the cache formula; watch the cache-size readout scale linearly with both.
                The colored streams show each query head continuously pulling from its shared KV head — every query still computes its own attention scores, only the K/V storage (and the memory bandwidth to read it) is shared.
              
              Real-world relevance: this exact mechanism is why modern open-weight LLMs can serve long conversations with far less GPU memory per request than an old-style MHA transformer of the same size — it trades a small amount of attention diversity for a large, near-linear cut in inference memory and bandwidth.

Query heads (cones) Shared KV heads (spheres)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.