Home▸AI & Machine Learning▸2D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax

🔠 2D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax

Token-tile rows for eight synthetic languages under one shared BPE vocabulary: corpus share, vocabulary size and word length decide tokens per word, how many words fit in a fixed window or context, and the relative cost tax, verified by round trips, an independent encoder and exhaustive-merge limits.

AI & Machine Learning2DModerate60 FPS📱 Mobile-adapted🟠 Blender⇄ 3D version
2D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax ↗ Open standalone
⚙ Under the hood

Token-tile rows for eight synthetic languages under one shared BPE vocabulary: corpus share, vocabulary size and word length decide tokens per word, how many words fit in a fixed window or context, and the relative cost tax, verified by round trips, an independent encoder and exhaustive-merge limits.

TokenizationByte-Pair EncodingFertilityMultilingual NLPZipf LawContext WindowVocabulary SizeLow-Resource Languages

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)