Transformer Attention Mechanism Explained
How multi-head self-attention lets transformer models like GPT weigh every word in a sentence against every other word, with temperature, positional bias and embedding dimension as tunable levers.
Why this matters
Self-attention is the core operation inside every modern large language model — GPT, BERT, Claude and the rest all stack layers built around it. Instead of processing a sentence strictly left to right like older recurrent networks, a transformer lets every token look directly at every other token in the sequence and decide, numerically, how much attention to pay to each one.
That single mechanism — score every pair, turn scores into a probability distribution with softmax, then blend value vectors by that distribution — is what lets transformers capture long-range dependencies (a pronoun referring back to a noun ten words earlier) as easily as adjacent-word relationships.
How the maths works
- Query, key and value projections — each input token's embedding (plus a positional encoding) is projected through learned matrices into a query vector, a key vector and a value vector.
- Scaled dot-product scores — every query is compared against every key with a dot product, then divided by the square root of the embedding dimension so the scores don't explode as dimensionality grows.
- Softmax normalisation — the raw scores for a given query are exponentiated and normalised so they sum to 1, turning them into an attention distribution over all tokens (including the token itself).
- Weighted sum of values — the output for each token is the value vectors of every other token, blended according to that attention distribution.
- Multiple heads — the whole process runs several times in parallel with separate learned projections ("heads"), so different heads can specialise in different relationships (e.g. syntax vs. coreference) before their outputs are concatenated.
Reading an attention heatmap
An attention matrix for one head is a grid where each row is a query token and each column is a key token; brightness encodes how strongly that row attends to that column. A sharp, near-diagonal pattern means tokens mostly attend to their neighbours; a spread-out row means the model is drawing information from many different parts of the sentence at once.
Two parameters change this shape a lot. Temperature divides the scores before softmax — turn it down and the distribution sharpens toward a single dominant token (low entropy, high sparsity); turn it up and attention spreads more evenly across the sequence. Positional bias adds a distance penalty so tokens further apart in the sentence get down-weighted by default, which is roughly what real trained position-aware attention heads learn to do on their own.
Why multiple heads help
A single attention head is a single view of the relationships in a sentence. Real transformers stack many heads — often 8, 12 or more per layer — and each one converges on a different pattern during training: some heads track adjacent-word syntax, some track long-range coreference, others track punctuation or sentence boundaries. The diversity metric (how different the heads' attention patterns are from each other, measured by cosine distance) is a useful proxy for how much unique information the layer is capturing rather than repeating the same signal eight times over.
Frequently Asked Questions
What is self-attention in simple terms?
It is a way for a model to decide, for every word in a sentence, how much weight to give to every other word when building that word's contextual representation — instead of only looking at nearby words in order.
Why is it called "multi-head" attention?
Because the attention calculation is run several times in parallel with different learned projections ("heads"), letting the model track several different kinds of relationships between tokens simultaneously, then combining the results.
What does the temperature parameter actually do?
It scales the raw attention scores before the softmax step. Lower temperature makes the resulting distribution sharper and more concentrated on a few tokens; higher temperature spreads attention more evenly across the whole sequence.
Why do transformers need positional encoding at all?
The core attention operation has no built-in sense of word order — it treats the input as an unordered set of tokens. Positional encodings inject information about each token's position in the sequence so the model can distinguish 'dog bites man' from 'man bites dog'.
Is this the same attention used inside GPT-style models?
Yes, conceptually — GPT-style decoder transformers use masked multi-head self-attention (each token can only attend to itself and earlier tokens), which is the same scaled dot-product mechanism with an additional causal mask applied before the softmax.