Rotary Position Embeddings (RoPE)

Modern transformer language models like LLaMA and Mistral don't add a positional signal to their embeddings — they rotate query and key vectors by an angle proportional to token position. This simulator renders every token in a sequence as a spoke spinning in its own 2D plane, lets you pick a query token and a key token to compare, and shows the resulting attention dot product live. Because rotating two vectors by mθ and nθ leaves their dot product dependent only on (m−n)θ, sliding the whole sequence window forward with the shift control leaves the highlighted score untouched even as both absolute angles keep advancing — the exact relative-position invariance that lets RoPE-based models extrapolate to context lengths they never trained on.