Home▸AI & Machine Learning▸Transformer Self-Attention: Query/Key/Value Heatmap (2D)

Transformer Self-Attention: Query/Key/Value Heatmap (2D)

2D self-attention lab: real query/key/value matrix projections, scaled dot-product scores and a live softmax heatmap over a token sequence — click a token to see exactly what it attends to.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Details

CategoryAI & Machine Learning

RenderingCanvas 2D

DifficultyAdvanced

Target FPS60

ControlsSet the token count and softmax temperature, click any token (or a heatmap row) to make it the active query, toggle raw pre-softmax scores, and reshuffle the embeddings and Q/K/V weight matrices to see a fresh attention pattern.

self-attentiontransformerquery key valuesoftmaxscaled dot-product attentionattention heatmap

Related simulations

Transformer Attention Mechanism ExplainedThe 3D version of this simulation.

Multi-Layer CNN Forward Pass: Conv → Pool → Dense → Softmax (2D)Another real-math 2D network lab — this one runs convolution and pooling instead of attention. ⊞ All in AI & Machine Learning →

This 2D companion runs the exact scaled-dot-product self-attention computation behind the 3D scene on a plain canvas built for reading the math rather than orbiting a scene: fixed token embeddings are projected through real query, key and value weight matrices, attention scores are computed as Q·Kᵀ divided by √d_k, and a genuine softmax turns them into a token-by-token attention-weight heatmap — click any token to make it the active query and see precisely which other tokens it attends to, with every row's weights verified to sum to 1.0 regardless of the temperature setting.

⚙ Under the hood

2D self-attention lab with real query/key/value matrix projections, scaled dot-product scores and a live softmax heatmap over an adjustable-length token sequence, recomputed on every parameter change.

self-attentiontransformerquery key valuesoftmaxscaled dot-product attentionattention heatmap

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install