🔠 2D BPE Tokenizer Fertility Gap: Multilingual Corpus Share, Vocabulary Size and the Token Tax
Token-tile rows for eight synthetic languages under one shared BPE vocabulary: corpus share, vocabulary size and word length decide tokens per word, how many words fit in a fixed window or context, and the relative cost tax, verified by round trips, an independent encoder and exhaustive-merge limits.
⚙ Under the hood
Token-tile rows for eight synthetic languages under one shared BPE vocabulary: corpus share, vocabulary size and word length decide tokens per word, how many words fit in a fixed window or context, and the relative cost tax, verified by round trips, an independent encoder and exhaustive-merge limits.
TokenizationByte-Pair EncodingFertilityMultilingual NLPZipf LawContext WindowVocabulary SizeLow-Resource Languages
2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install