The simulation visualizes sinusoidal positional encoding vectors as glowing wave patterns across embedding dimensions, showing how each position in a sequence gets a distinct combination of sine and cosine frequencies.
Use the sequence length slider to change how many word positions are shown and the speed slider to control animation pace, then press play to watch the encoding patterns build position by position, or rebuild to regenerate the scene.
Sequence length slider, speed slider, play/pause, rebuild
In the original transformer paper, the wavelengths of the sine and cosine functions used for positional encoding form a geometric progression from 2π up to roughly 10,000 times 2π, letting the model represent both fine-grained and very coarse positional differences at once.
The simulation visualizes sinusoidal positional encoding vectors as glowing wave patterns across embedding dimensions, showing how each position in a sequence gets a distinct combination of sine and cosine frequencies.
The simulation visualizes sinusoidal positional encoding vectors as glowing wave patterns across embedding dimensions, showing how each position in a sequence gets a distinct combination of sine and cosine frequencies.
Use the sequence length slider to change how many word positions are shown and the speed slider to control animation pace, then press play to watch the encoding patterns build position by position, or rebuild to regenerate the scene.
In the original transformer paper, the wavelengths of the sine and cosine functions used for positional encoding form a geometric progression from 2π up to roughly 10,000 times 2π, letting the model represent both fine-grained and very coarse positional differences at once.