Home▸Machine Learning & Neural Networks▸Byte Pair Encoding: How Tokenizers Learn Subwords

🧩 Byte Pair Encoding: How Tokenizers Learn Subwords

Explore how Byte Pair Encoding builds a vocabulary of subword units by repeatedly merging the most frequent adjacent symbol pairs, the algorithm behind tokenizers in GPT, RoBERTa and many other modern language models.

Machine Learning & Neural Networks3DModerate60 FPS

The simulation shows a small training corpus of words broken into characters, then animates BPE's merge loop: counting adjacent pair frequencies, highlighting the most frequent pair, merging it into a new symbol, and updating the growing vocabulary in real time.

🔬 What It Demonstrates

🎮 How to Use

Choose a sample corpus, then press play to step through merge iterations one at a time and watch the vocabulary and tokenized words update, or use the speed slider to run many merges quickly and the rebuild button to start over with fresh word frequencies.

💡 Did You Know?

GPT-2's byte-level BPE tokenizer starts from a base alphabet of just 256 possible byte values and learns roughly 50,000 merges, meaning the vast majority of its 50,257-token vocabulary is subwords the algorithm discovered purely from text frequency, not hand-picked by a linguist.

⚙ Under the hood

Watch the most frequent adjacent character pair merge into a new symbol over and over, building a subword vocabulary from raw text.

byte pair encodingbpetokenizationsubwordnlpmachine-learning

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install