🔠 3D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax
Eight Blender-built shelves, one per synthetic language, hold the tokens of a real frequency-weighted BPE tokenizer trained on a Zipf corpus: change the corpus share of a language or the vocabulary size and watch tiles re-merge, fertility (tokens per word) and the cost tax change, checked by decode round trips, an independent encoder and monotonicity laws.
Eight Blender-built shelves, one per synthetic language, hold the tokens of a real frequency-weighted BPE tokenizer trained on a Zipf corpus: change the corpus share of a language or the vocabulary size and watch tiles re-merge, fertility (tokens per word) and the cost tax change, checked by decode round trips, an independent encoder and monotonicity laws.
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install