Home▸Machine Learning & Neural Networks▸Positional Encoding: How Transformers Know Word Order

🧭 Positional Encoding: How Transformers Know Word Order

Transformers process every word at once, so how do they know 'dog bites man' differs from 'man bites dog'? Explore the sine-and-cosine trick that injects word order into attention.

Machine Learning & Neural Networks3DModerate60 FPS

The simulation visualizes sinusoidal positional encoding vectors as glowing wave patterns across embedding dimensions, showing how each position in a sequence gets a distinct combination of sine and cosine frequencies.

🔬 What It Demonstrates

🎮 How to Use

Use the sequence length slider to change how many word positions are shown and the speed slider to control animation pace, then press play to watch the encoding patterns build position by position, or rebuild to regenerate the scene.

💡 Did You Know?

In the original transformer paper, the wavelengths of the sine and cosine functions used for positional encoding form a geometric progression from 2π up to roughly 10,000 times 2π, letting the model represent both fine-grained and very coarse positional differences at once.

⚙ Under the hood

See how sinusoidal positional encodings at different frequencies combine to give every position in a sequence a unique, smoothly varying signature.

positional encodingtransformersattentionnlpmachine-learningsequence models

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install