Scaled Dot-Product Attention: The Role of √d_k

This simulator isolates one specific mechanic inside every transformer attention head: the √d_k scaling term in softmax(QKᵀ/√d_k)V. Nine tokens sit in a 3D row, each holding a random query and key vector; picking a query token computes its raw dot product against every key, optionally divides by the square root of the key dimension, applies a softmax, and renders the resulting attention distribution as glowing beams and flowing particles into the query token, with bars above each token tracking its live weight. Widening the key dimension d_k with scaling switched off drives the raw scores' standard deviation up and collapses the softmax onto a single token — exactly the vanishing-gradient failure mode the √d_k term exists to prevent — while switching scaling back on keeps the distribution well-behaved at any width.