Grouped-Query Attention: Shrinking the KV Cache

Attention Configuration

Query heads (H) 32

KV head groups (G) 8

Sequence length cached 4096 tok

Live Readouts

Heads per KV group4 : 1

KV cache size (this run)—

Memory saved vs full MHA—

MQA / MHA ratio (verify)—

How it works

In standard multi-head attention (MHA) every one of the H query heads keeps its own key and value projection, so the KV cache — the running memory of every past token's K and V vectors that autoregressive decoding must keep around — grows with H. Grouped-Query Attention (GQA) splits the H query heads into G groups and gives every head in a group the same shared K and V head, cutting the cache by a factor of H⁄G while every query head still attends independently:

KV cache bytes = G × S × d_head × 2 × b
  G = number of KV head groups, S = cached sequence length,
  d_head = per-head dimension, b = bytes per stored value (2 for fp16/bf16)
  factor 2 = one tensor for K, one for V

G = H  → MHA  (no sharing, largest cache, highest quality)
G = 1  → MQA  (all heads share one KV pair, smallest cache)
1
              
                Query heads (H) / KV groups (G) — choose any G that divides H; the diagram regroups the query-head blocks (top row) around the shared KV-cache blocks (bottom row, one per group) live.
                Preset buttons — jump straight to the three named regimes for the current H.
                Sequence length — the other multiplier in the cache formula; watch the cache-size readout scale linearly with it too.
                The chart below the diagram sweeps every valid group count from 1 (MQA) to H (MHA) and plots the resulting cache size, showing the exact linear relationship — doubling G exactly doubles the cache.
              
              Real-world relevance: this exact mechanism is why modern open-weight LLMs can serve long conversations with far less GPU memory per request than an old-style MHA transformer of the same size — it trades a small amount of attention diversity for a large, near-linear cut in inference memory and bandwidth.

Query heads Shared KV-cache blocks