Positional Encoding: How Transformers Know Word Order

The Problem: Parallel Processing Has No Sense of Order

Older sequence models like RNNs read text one word after another, so order is baked into how they process information step by step. Transformers abandoned that step-by-step approach for speed, letting every token attend to every other token simultaneously — but that parallelism comes at a cost. Stripped of any extra signal, the core attention mechanism treats a sentence as an unordered set of tokens, meaning 'the cat sat on the mat' and a scrambled version like 'mat the on sat cat the' would look identical to it. Something has to explicitly tell the model which word came first, second, and so on.

The Solution: Adding a Positional Signature

The fix used in the original transformer paper is to compute a positional encoding vector for every position in the sequence and simply add it to that token's word embedding before anything enters the network. Each positional vector is built from sine and cosine waves of many different frequencies, one pair per dimension of the embedding, so position 0 gets one distinctive pattern of values, position 1 gets a slightly different one, and so on. Because the frequencies range from very fast-oscillating to very slow-oscillating across the dimensions, every position ends up with a unique, smoothly varying signature that the model can learn to read like a coordinate.

Why Sine and Cosine? The Mathematical Payoff

Sinusoidal functions were not an arbitrary choice: a key property is that the encoding for any position p+k can be written as a simple linear transformation of the encoding for position p, for any fixed offset k. That means the network can more easily learn to attend based on relative distance between words, such as 'the word two positions back', rather than only memorizing absolute positions. As a bonus, because sine and cosine are smooth and periodic, the scheme extrapolates reasonably well to sequence lengths longer than anything seen during training, unlike a lookup table of positions the model has never encountered.

Beyond the Original Paper: Positional Encoding Today

The sinusoidal scheme from Vaswani et al.'s 2017 'Attention Is All You Need' remains a beautifully simple and instructive baseline, but it is no longer the only game in town. Many modern large language models use alternatives such as learned positional embeddings trained directly from data, or rotary position embeddings (RoPE), which rotate query and key vectors in a way that encodes relative position directly inside the attention calculation itself. Whatever the specific mechanism, the underlying job is identical: give an architecture that is inherently blind to sequence order a reliable way to tell first from last.

Frequently asked questions

Try it live

Everything above runs in your browser — open Positional Encoding: How Transformers Know Word Order and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.