Self-Attention Simulator | mysimulator.uk

Self-Attention

Sentence

Temperature (softmax) 1.0

Key noise 0.10

Animation speed 1.0×

Entropy

–

Top pair

–

Max weight

–

Tokens

–

How it works

Every token in the sentence is projected into a query vector (what it's looking for) and a key vector (what it offers). The score between token i and token j is the dot product of query_i and key_j, scaled by the square root of the vector dimension so the values stay in a stable range. Running softmax over each row turns those raw scores into a probability distribution — the attention weights — which says how much token i should "listen" to every other token. Each token's new representation is the weighted sum of everyone's value vectors, using its own row of weights.

score(i,j) = (qᵢ · kⱼ) / √d
weight(i,j) = softmax_j( score(i,j) / T )
outᵢ = Σⱼ weight(i,j) · vⱼ

Sentence — swap between sentences where the same word ("bank") needs different context to disambiguate, showing why static word vectors alone aren't enough.
Focus token — click a token in the heatmap or use the button to isolate one row and see exactly which other words it attends to.
Temperature — divides scores before softmax; low temperature sharpens attention onto the single best match, high temperature spreads it almost uniformly.
Key noise — perturbs the key vectors each frame, showing how attention weights are sensitive to how well-separated the learned representations are.

Real-world relevance: this row-normalized weighted sum is the core operation inside every transformer layer (BERT, GPT) — stacking many attention heads and layers is how models like these build up contextual understanding of a sentence, exactly the "self-attention" concept the article describes.

High attention Low attention Focused token