The simulation shows a small training corpus of words broken into characters, then animates BPE's merge loop: counting adjacent pair frequencies, highlighting the most frequent pair, merging it into a new symbol, and updating the growing vocabulary in real time.
Choose a sample corpus, then press play to step through merge iterations one at a time and watch the vocabulary and tokenized words update, or use the speed slider to run many merges quickly and the rebuild button to start over with fresh word frequencies.
Corpus selector, speed slider, play/pause, step-forward, rebuild vocabulary
GPT-2's byte-level BPE tokenizer starts from a base alphabet of just 256 possible byte values and learns roughly 50,000 merges, meaning the vast majority of its 50,257-token vocabulary is subwords the algorithm discovered purely from text frequency, not hand-picked by a linguist.
The simulation shows a small training corpus of words broken into characters, then animates BPE's merge loop: counting adjacent pair frequencies, highlighting the most frequent pair, merging it into a new symbol, and updating the growing vocabulary in real time.
The simulation shows a small training corpus of words broken into characters, then animates BPE's merge loop: counting adjacent pair frequencies, highlighting the most frequent pair, merging it into a new symbol, and updating the growing vocabulary in real time.
Choose a sample corpus, then press play to step through merge iterations one at a time and watch the vocabulary and tokenized words update, or use the speed slider to run many merges quickly and the rebuild button to start over with fresh word frequencies.
GPT-2's byte-level BPE tokenizer starts from a base alphabet of just 256 possible byte values and learns roughly 50,000 merges, meaning the vast majority of its 50,257-token vocabulary is subwords the algorithm discovered purely from text frequency, not hand-picked by a linguist.