Every token position computes a Query, Key and Value vector. Each head scores how much every other token's Key matches its Query, turns those scores into a probability distribution with softmax, and mixes the Values by that distribution — the token "looks at" the rest of the sequence.
Attention(Q,K,V) = softmax(QK^T / √d_k) · V
- Sequence length — how many tokens sit on each layer's ring.
- Attention heads — parallel attention computations (colour-coded), each free to focus on a different pattern.
- Encoder layers — stacked self-attention + feed-forward blocks; thin vertical lines are residual/skip connections carrying each token upward.
- Temperature — scales the softmax logits: low temperature sharpens attention onto a few tokens, high temperature spreads it almost uniformly.
This is the mechanism behind BERT, GPT and T5: parallel, all-pairs attention instead of the sequential recurrence of RNNs/LSTMs.