Every token gets a learned query vector (what it's looking
for) and a key vector (what it offers). The query token
(glowing white, pulsing) compares its query against every other
token's key with a dot product, scales by 1/√dk, and
turns the scores into a probability distribution with softmax —
that's the attention weight drawn as a curved beam to each key.
Brighter, thicker beams mean more attention. Raising the
temperature flattens the distribution (attention spreads
thin, high entropy); lowering it sharpens focus onto one or two
tokens (low entropy). Multiple heads run this whole process in
parallel with independent query/key projections — each head can
learn to track a different kind of relationship, shown here as
differently-coloured beam sets.
Attention(Q, K, V) = softmax(QK^T / √d_k) V
score_ij = (q_i · k_j) / √d_k
weight_ij = softmax_j(score_ij / T)
- Tokens — length of the input sequence being attended over.
- Temperature (T) — divides the scores before softmax; low T sharpens focus, high T spreads attention almost uniformly.
- Heads — parallel attention computations with independent Q/K projections, each drawn in its own colour.
- Auto-cycle — steps the query token every couple of seconds; switch to Manual and click a sphere to inspect one token's attention pattern.
This is the mechanism that replaced the RNN "bottleneck" vector
(one fixed-size summary for the whole sequence) with direct,
weighted access to every token — the core idea behind
self-attention and cross-attention in the Transformer
architecture used by modern language and vision models.