Transformers process a sentence not word-by-word in sequence, but all at once — every token can directly look at every other token through self-attention. Each token is projected into a query (what am I looking for?), a key (what do I offer?), and a value (what information do I carry?). A token's new representation is a weighted blend of every other token's value vector, where the weights come from comparing its query against every key.
i and key token j, the raw score is score(i,j) = Q_i · K_j / √d.softmax(score / T) — the arcs in the scene are colored, weighted beams from the selected query token to every other token, one color per attention head.The 2017 paper "Attention Is All You Need" removed recurrence entirely, letting every token attend to every other token in a single matrix multiplication — the key idea that made today's large language models trainable at scale.
Six tokens sit in a ring; glowing beams show how much each token's query attends to every other token's key, with brightness and thickness mapped to the softmax attention weight for each head.
Scaled dot-product attention: scores from Q·K are divided by √d, passed through a temperature-scaled softmax, and used to blend value vectors from every token into the selected query token.
Pick a query token and how many heads to compute. Lower the temperature to see attention sharpen onto one word, or raise it to flatten it toward uniform. Isolate a single head or overlay them all, and watch value particles flow in along each beam.
Multi-head attention lets a transformer look for several kinds of relationships at once — one head might track subject-verb agreement while another tracks nearby adjectives, all computed in parallel.