Transformer Self-Attention: Query/Key/Value Heatmap (2D)

Sequence

Tokens in sequence 6

Softmax

Temperature (sharpness) 1.00

Selected query token

Live Readout

Top attends to

—

Max weight

—

Row Σ (must be 1)

—

Entropy

—

How it works

Every token starts as a fixed d=8 embedding vector. Three small weight matrices project each embedding into a query, key and value vector (d_k = 4):

Q_i = E_i · W_Q     K_j = E_j · W_K     V_j = E_j · W_V
score(i,j) = (Q_i · K_j) / √d_k
weight(i,:) = softmax( score(i,:) / T )
output_i = Σ_j weight(i,j) · V_j

The heatmap is weight(i,j) for every query row i against every key column j — real linear algebra recomputed on every slider move, not a decorative overlay. Each row is a genuine probability distribution: it always sums to 1.0 regardless of temperature, because softmax normalizes it — T only reshapes how peaked or flat that distribution is before normalization.

Click a token (top strip or picker) to make it the active query and see its full attention row.
Temperature ↓ — scores are divided by a smaller T, so softmax sharpens toward the single best-matching key.
Temperature ↑ — scores shrink toward 0, softmax flattens toward uniform 1/N attention.
Raw scores toggle — shows the unnormalized Q·K/√d_k values (can be negative) instead of the softmax weights, so you can see what softmax is normalizing.

Attention weight (bright = high) Selected query row