Home▸AI & Machine Learning▸3D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax

🔠 3D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax

Eight Blender-built shelves, one per synthetic language, hold the tokens of a real frequency-weighted BPE tokenizer trained on a Zipf corpus: change the corpus share of a language or the vocabulary size and watch tiles re-merge, fertility (tokens per word) and the cost tax change, checked by decode round trips, an independent encoder and monotonicity laws.

AI & Machine Learning3DModerate60 FPS📱 Mobile-adapted🟠 Blender⇄ 2D version
3D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax ↗ Open standalone
⚙ Under the hood

Eight Blender-built shelves, one per synthetic language, hold the tokens of a real frequency-weighted BPE tokenizer trained on a Zipf corpus: change the corpus share of a language or the vocabulary size and watch tiles re-merge, fertility (tokens per word) and the cost tax change, checked by decode round trips, an independent encoder and monotonicity laws.

TokenizationByte-Pair EncodingFertilityMultilingual NLPZipf LawContext WindowVocabulary SizeLow-Resource Languages

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)