This is a 2D matrix-and-distribution view of transformer self-attention:
Attention(Q,K,V) = softmax( QKᵀ / √d_k ) V
The left grid is the full 9×9 attention matrix — every query token (rows) against every key token (columns), colored by softmax weight. The highlighted row is the query token picked below. The right panel is a live histogram of 4,000 freshly-sampled independent-Gaussian dot products at the current d_k, with the theoretical Normal(0, d_k) curve for their variance drawn on top — this is a direct, numerically-checkable picture of "variance grows linearly with d_k" rather than a single 3D beam view of one query row.
If each of the d_k components of Q and K is an independent random variable with mean 0 and variance 1, their dot product is a sum of d_k independent terms, so its variance grows linearly with d_k (std dev grows like √d_k). Dividing every score by √d_k rescales the standard deviation back to roughly 1 regardless of dimension, which is why the histogram narrows back to a fixed width once scaling is switched on, and why the attention matrix stops collapsing to a single hot column per row.
- d_k slider — widens Q/K vectors; watch the raw-score histogram spread toward std ≈ √d_k when scaling is off.
- √d_k scaling toggle — turn it off to watch the attention matrix collapse to near-one-hot rows (entropy → 0) as d_k grows.
- Temperature T — extra softmax sharpening/softening knob (logits divided by T), independent of the dimension-scaling fix.
- Query token — picks which matrix row (and the stat readouts) is being inspected.
This is the exact scaling term from "Attention Is All You Need" (Vaswani et al., 2017) — every dot-product-attention layer in GPT, BERT, and every modern transformer uses it.