Home▸AI & Machine Learning▸Scaled Dot-Product Attention (2D): Score Matrix & √d_k Distribution

Scaled Dot-Product Attention (2D): Score Matrix & √d_k Distribution

2D attention-matrix and score-distribution visualization of the √d_k scaling term in transformer attention: watch the full 9×9 softmax heatmap and the empirical vs theoretical raw-score histogram as key dimension and scaling change.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

This 2D companion isolates the same √d_k scaling mechanic as the 3D version, but represents it as two native-2D fields instead of a rotatable 3D beam scene: a full 9×9 query-by-key attention heatmap, and a live histogram of the raw dot-product score distribution overlaid with its theoretical Normal(0, d_k) curve. Widening the key dimension d_k with scaling switched off visibly spreads the histogram and collapses matrix rows onto a single bright cell — exactly the vanishing-gradient failure mode the √d_k term exists to prevent — while switching scaling back on keeps both the histogram width and the matrix's row-wise weight distribution well-behaved at any dimension.

⚙ Under the hood

2D matrix-and-histogram visualization of the √d_k scaling term in transformer attention: watch the full 9×9 softmax attention heatmap and a live histogram of raw query-key dot-product scores against their theoretical Normal(0, d_k) curve as key dimension, scaling, and temperature change.

transformersattentionsoftmaxself-attentiondeep-learningneural-networks2d

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install