Self-Attention: Scaled Dot-Product Visualizer

Sequence Length

Query Token

Q: "the"

Temperature (softmax sharpness) 1.00×

Attention Weights (this query)

Max weight

--%

Entropy

-- bits

Effective divisor √d·T

2.83

Full Attention Matrix

key →query ↓

How it works

Each token starts as an embedding vector. Three learned projection matrices turn it into a Query, a Key and a Value:

Q = X·Wq   K = X·Wk   V = X·Wv
scores  = Q·Kᵀ / (√d · T)
weights = softmax(scores)   (per row)
output  = weights · V

√d (d = embedding dimension) keeps dot-product magnitudes from exploding as d grows; the temperature slider T scales that divisor further — low T sharpens the softmax toward a single dominant key, high T spreads attention almost uniformly across all tokens.

Sequence length — how many tokens sit in the circle; every token attends to every other, including itself.
Query token — which row of the attention matrix is drawn as 3D beams and highlighted in the heatmap.
Temperature — rescales the score divisor; watch the beams collapse onto one key at low T and fan out evenly at high T.
Randomize embeddings — draws a new random embedding for every token (and re-projects Q/K/V), the way a new input sentence would.

This is the exact mechanism (scaled dot-product self-attention) at the core of every transformer — GPT, BERT, ViT — repeated across many heads and layers to build contextual representations.

Beams = attention from the highlighted query token to every key. Thicker & redder = higher weight.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.