Zipf's law says the frequency of the r-th most common word in a language falls off as a power law of its rank:
f(r) ≈ C / r^s
log f(r) = log C − s·log r
In log-log axes that is a straight line with slope −s. For natural language s is close to 1: the most frequent word ("the" in English) occurs roughly twice as often as the second-ranked word, three times as often as the third, and so on.
- Synthetic corpus — builds a discrete probability table p(r) ∝ 1/r^s over a vocabulary of V "words" (normalised so it sums to 1), then draws N tokens from it by inverse-transform sampling (binary search on the CDF). The teal marks are the resulting word counts, re-sorted by observed frequency, showing that sampling noise still lands on the same log-log line.
- Vocabulary size V and corpus size N control how far the tail extends and how noisy the low-frequency ranks look; a small N relative to V leaves many words unseen (0 count, so absent from the plot).
- Exponent s steepens or flattens the curve — s → 0 approaches a uniform distribution, s ≳ 1.5 makes a handful of words dominate almost the whole corpus.
- Real text overlay — the amber marks tokenize whatever text sits in the box, count word frequencies, and fit the same log-log line live. Real English text consistently lands near s ≈ 1.
- The red dashed line on each series is the least-squares fit on (log rank, log frequency); drag the main plot to pan and scroll/pinch to zoom into any region of it.
- The lower panel is the Lorenz-style cumulative coverage curve: x = fraction of distinct words seen (by rank), y = fraction of all tokens they account for. A more skewed (bowed) curve means a smaller core vocabulary covers most of the text — directly reading off "words needed for 50% of tokens" above.
Real-world relevance: Zipf's law is why stop-word lists exist (a tiny set of words covers most of any corpus), why byte-pair/subword tokenizers spend their merge budget on the most frequent character pairs first, and why naive frequency-based compression schemes can get most of their gain from a handful of common tokens.