Byte Pair Encoding: How Tokenizers Learn Subwords

The Trouble With Whole Words

The most obvious way to tokenize text is to split on spaces and treat every distinct word as a unit in the vocabulary. This works fine until the model meets a word it has never seen before. Natural language is full of rare surnames, typos, invented brand names, technical jargon, and words in other languages, and a fixed word-level vocabulary has no way to represent any of them except a generic, information-destroying <UNK> token. Worse, a word-level vocabulary treats "run", "runs", "running", and "runner" as four completely unrelated symbols, so the model gets no head start on the fact that they share a common root. To cover even a modest fraction of the words that might plausibly appear, a word-level vocabulary would need to be enormous, and it would still be caught off guard the first time someone typos a word or invents a new one.

The Trouble With Single Characters

The opposite extreme is to tokenize text one character at a time. This solves the coverage problem completely, since any string can be built from a small alphabet of characters, and there is no such thing as an out-of-vocabulary character. The cost is length: a sentence that might have been 8 or 10 word tokens now becomes 40 or 50 character tokens. Because transformer models process every token in a sequence and their computational cost grows with sequence length, character-level tokenization makes training and inference dramatically more expensive, and it also forces the model to work much harder to reconstruct meaning, since a single character like "t" carries almost no information on its own. Somewhere between one gigantic vocabulary of whole words and one tiny vocabulary of individual characters lies a sweet spot, and that is exactly what Byte Pair Encoding is built to find.

Learning Merges From Frequency

Byte Pair Encoding starts every word in the training corpus broken down into its individual characters, so "lower" begins life as the five symbols l, o, w, e, r. The algorithm then counts every adjacent pair of symbols across the entire corpus, and finds the single pair that occurs most often, say the pair (l, o) if "low", "lower", and "lowest" are all common in the training text. That most-frequent pair is merged into a brand-new symbol, "lo", which is added to the vocabulary, and every occurrence of that pair anywhere in the corpus is replaced with the merged token. The corpus is then rescanned with this new symbol in place, pair frequencies are recounted, and the next most frequent pair is merged. Each merge is recorded in order, because that order matters later: when the trained tokenizer encounters new text, it applies the same merges in the same sequence to decide how to split it.

From Characters to Common Subwords

Repeating this merge step thousands of times is where the magic compounds. Early merges tend to combine common character pairs into small chunks like "in", "er", or "th". Later merges combine those chunks into larger units, and sufficiently frequent whole words like "the", "and", or "tokenization" can eventually end up as a single merged symbol in the vocabulary, exactly like a word-level tokenizer would produce for common words. The number of merge operations performed is a hyperparameter chosen before training, typically tens of thousands, and it directly controls the final vocabulary size: stop early and you get more, shorter subwords; run more merges and you get fewer, longer ones. Crucially, rare or novel words never get their own dedicated merge, so they simply fall back to being spelled out from smaller, already-learned pieces, like "tokenization" splitting into "token" and "ization" if the whole word itself was not frequent enough to earn its own merge.

The Backbone of Modern Tokenizers

This is precisely why GPT-style models, RoBERTa, and many other transformers can handle typos, rare technical terms, emoji, and text in languages barely represented in training, without ever emitting an unknown-token placeholder. GPT-2 and its successors apply BPE directly over raw bytes rather than Unicode characters, which guarantees that absolutely any input, including unusual symbols or mixed scripts, can always be represented using a fixed base alphabet of 256 byte values plus the learned merges. The resulting vocabulary, typically containing 30,000 to 100,000 subword tokens, strikes a practical balance: common words and word fragments are single efficient tokens, while anything unfamiliar gracefully degrades into smaller, still-meaningful pieces rather than breaking the model entirely. That balance between vocabulary size, sequence length, and universal coverage is why BPE and its close relatives, like WordPiece and Unigram tokenization, remain the standard first step in nearly every modern large language model pipeline.

Frequently asked questions

Try it live

Everything above runs in your browser — open Byte Pair Encoding: How Tokenizers Learn Subwords and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.