Scaled Dot-Product Attention: The Role of √d_k

Attention Head Controls

Key/query dimension d_k 64

Apply √d_k scaling

Softmax temperature T 1.00

Query token #4

Live Readouts

Raw score std dev

–

Max attn weight

–

Softmax entropy (max log₂9 = 3.17 bits)

–

score = Q·K / √d_k

How it works

Transformer self-attention scores every key against a query with a dot product, then normalizes with softmax:

Attention(Q,K,V) = softmax( QKᵀ / √d_k ) V

If each of the d_k components of Q and K is an independent random variable with mean 0 and variance 1, their dot product Q·K is a sum of d_k independent terms, so its variance grows linearly with d_k (std dev grows like √d_k). Without correction, raw scores for a 64- or 256-dimensional head can reach magnitudes of ±8–±16.

Softmax is exponential: e^(score). Feeding it scores that large pushes it into a saturated regime where one key gets ≈100% of the weight and every gradient through the other keys vanishes — the network stops learning which keys matter. Dividing every score by √d_k rescales the standard deviation back to roughly 1 regardless of dimension, keeping softmax in its well-conditioned range no matter how wide the head is.

d_k slider — widens the query/key vectors; watch the raw-score std dev climb toward √d_k when scaling is off.
√d_k scaling toggle — turn it off to watch attention collapse onto a single token (entropy → 0, max weight → 100%) as d_k grows.
Temperature T — an extra softmax sharpening/softening knob (logits divided by T) — the same mechanism used to control sampling diversity at inference time, independent of the dimension-scaling fix.
Query token — picks which token's Q vector is being compared against every K vector; beams and particle flow always point into this token.

This is the exact scaling term from "Attention Is All You Need" (Vaswani et al., 2017) — every dot-product-attention layer in GPT, BERT, and every modern transformer uses it.

Query token Key tokens · beam = attention weight

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.