Rotary Position Embeddings (RoPE)

Sequence & Tokens

Sequence length N 12

Query token (m) 3

Key token (n) 8

Rotation

Frequency band i 0

Window shift s 0

Live Readout

θ = 1.000 rad/token

Relative distance Δ=m−n

−5

Attention score cos(Δθ)

0.28

Query abs. angle

180°

Key abs. angle

459°

How it works

RoPE encodes the position of a token by rotating pairs of dimensions in its query/key vectors, instead of adding a positional signal. For dimension pair i at position m, with frequency θ_i = base^−2i/d:

R(mθ) = [ cos(mθ)  −sin(mθ) ]
        [ sin(mθ)   cos(mθ) ]

q_m = R(mθ) · q      (rotate the query by mθ)
k_n = R(nθ) · k      (rotate the key   by nθ)

Because rotation matrices compose as R(mθ)^TR(nθ) = R((n−m)θ), the dot product between a rotated query and a rotated key depends only on m−n, never on m or n individually:

q_m · k_n = |q||k| · cos((m−n)θ)

Query / Key sliders — pick which two tokens' vectors are highlighted (blue = query, orange = key); every token's spoke still rotates by its own position × θ.
Frequency band i — real transformers use d/2 bands, from fast-rotating (i=0) to nearly static (i large); low bands resolve nearby tokens, high bands carry coarse long-range position.
Window shift s — adds the same offset to every token's position, as if you slid a fixed-size attention window forward in a long sequence. Watch the score stay fixed while both absolute angles keep climbing — this is exactly why RoPE generalizes to sequence lengths never seen in training.

Real-world relevance: RoPE is the positional scheme used in LLaMA, Mistral, Qwen, GPT-NeoX and most modern open-weight language models, precisely because relative-position invariance lets attention patterns learned at short range transfer to longer contexts.

Query vector (m) Key vector (n) Other tokens Score link

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.