Home▸AI & Machine Learning▸Tokenizer Fertility Gap (2D): The Multilingual Vocabulary Tax

Tokenizer Fertility Gap (2D): The Multilingual Vocabulary Tax

Interactive 2D bar-chart visualizer: watch how a shared subword vocabulary (BPE / SentencePiece, as in mBERT, XLM-R, mT5) splits the same sentence into wildly different numbers of tokens depending on a language's share of training data and morphology — the 'fertility gap' that taxes low-resource languages with more compute and less usable context.

AI & Machine Learning2DModerate60 FPS📱 Mobile-adapted⇄ 3D version

Details

CategoryAI & Machine Learning

RenderingCanvas 2D

DifficultyModerate

Target FPS60

ControlsVocabulary-size slider sets the shared subword budget (now genuinely bends the fertility exponent); language dropdown picks the focus language; training-data-share slider raises or lowers that language's corpus fraction; sentence-length slider sets word count; 'Tokenize a sample word' button animates one word splitting into its subword fragments; drag the chart to pan across all 8 language columns; scroll/pinch over it to rescale bar height.

tokenizationmultilingualNLPBPElow-resource languagesLLM cost

Related simulations

Byte Pair Encoding: How Tokenizers Learn SubwordsWatch the most frequent adjacent character pair merge into a new symbol over and over,…

Multilingual & Cross-Lingual Models: A Shared Embedding SpaceWatch word embeddings from several languages sit in one shared vector space. Tune…

Stemming vs Lemmatization: Rule-Based Suffix StrippingStep through the real Porter stemming algorithm as it strips suffixes off a word and… ⊞ All 1161 in AI & Machine Learning →

Every multilingual language model tokenizes text with one shared subword vocabulary, trained once on a corpus dominated by a handful of high-resource languages. This 2D build renders that vocabulary as eight animated bar-chart columns of stacked token blocks with a live context-window gauge: pick a focus language, drag its share of the training corpus up or down, and watch how many token-blocks the same sentence needs — the "fertility" of the tokenizer for that language. Low-resource and morphologically rich languages routinely need several times as many tokens as English for identical meaning, which quietly taxes them with higher API cost per sentence and less usable context inside a fixed token window.

⚙ Under the hood

Watch a shared subword vocabulary (BPE / SentencePiece, as in mBERT, XLM-R, mT5) split the same sentence into wildly different numbers of tokens depending on a language's share of the training corpus, rendered as an animated 2D bar chart of stacked token blocks with a live context-window gauge. Drag the chart to pan across all 8 languages, scroll to rescale, and drag a language's data share and vocabulary size to see the fertility gap that taxes low-resource languages with more tokens, more compute cost, and less usable context window.

tokenizationmultilingualNLPBPElow-resource languagesLLM cost

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install