Multilingual models (mBERT, XLM-R, mT5, BLOOM, NLLB) share one subword vocabulary — built once with BPE or SentencePiece — across every language they support. The vocabulary's merge budget is spent where the training corpus is densest, so high-resource languages get many whole-word and long-subword tokens while low-resource languages are left splitting words into short fragments, sometimes down to single bytes.
This simulation's fertility model (2D build — vocabulary size is wired into the exponent below; the sibling 3D build's "Vocabulary size" slider updates its own on-screen number but was never actually read by its fertility formula, so dragging it there does nothing to the bars — fixed here, see comment in the source):
exponent(V) = 0.42 × (64000 / V) ^ 0.12
fertility(L) = morph(L) × (refShare / max(share(L), floor)) ^ exponent(V)
tokens(L) = words × fertility(L)
window_used = tokens(L) / 4096
- morph(L) — a per-language base multiplier for morphological complexity (analytic languages like English or Vietnamese need fewer subwords per word; agglutinative/templatic languages like Swahili, Turkish or Amharic need more even with a fair vocabulary share).
- share(L) — the language's fraction of the pretraining corpus (English dominates most web-scale corpora; many of the world's ~7,000 languages get well under 0.1%).
- refShare — English's own corpus share, used as the reference point where fertility ≈ morph(L).
- exponent(V) — a bigger shared vocabulary gives every language a few more merge slots, which mildly compresses (but never removes) the exponent that drives the gap; going from 8,000 to 256,000 merges moves the exponent from ≈0.53 down to ≈0.36 — noticeably softer bars, same underlying inequity.
The exponent and floor are tuned so the curve reproduces the qualitative shape reported by tokenizer-fairness studies (Rust et al. 2021, "How Good is Your Tokenizer?"; Ahia et al. 2023, "Do All Languages Cost the Same?") — under-resourced languages routinely need 3–10× more tokens than English for the same sentence. Because the tokenizer sees more, shorter pieces for these languages, users pay more API cost per sentence, and the model's fixed context window holds proportionally fewer real words of that language — the "fertility gap" is a hidden inequity in every multilingual LLM.
- Vocabulary size — helps everyone a little, but does not fix an unequal split of that vocabulary across languages.
- Training-data share — drag the focus language toward "more data" and watch its bar shrink toward English-level fertility; drag it toward "less data" and watch tokens fragment.
- Drag the chart to pan across all 8 languages; scroll/pinch over it to rescale bar height.