N-Gram Language Model: Live Perplexity & Text Generation

Training corpus

Sampling temperature 1.0

Add-k smoothing 0.10

Generation speed 1.5 tok/s

Generated so far

<s>

Metrics

Live perplexity

–

Held-out perplexity

–

Vocabulary size / tokens generated

–

How it works

The model is a real bigram language model: it counts every word pair (prev → next) in the training corpus and turns those counts into probabilities with add-k (Laplace) smoothing, so even pairs never seen in training get a small non-zero probability instead of breaking the model.

P(next|prev) = ( count(prev,next) + k ) / ( count(prev) + k·V )
perplexity   = exp( − (1/N) · Σ log P(wᵢ | wᵢ₋₁) )

Training corpus — switch between two small themed corpora; the model, vocabulary graph and both perplexity scores rebuild from scratch each time.
Sampling temperature — reshapes the generation distribution (Pⁱᐟᵀ, renormalized) without touching the perplexity math: low values make generation greedy and repetitive, high values make it wander.
Add-k smoothing — the Laplace constant k added to every count; push it up and the model flattens toward uniform, which raises live perplexity but can lower held-out perplexity on unseen word pairs.
Live perplexity — computed from the actual model probability (not the temperature-warped one) of every word the walker has generated so far — the standard measure of how "surprised" the model is by its own output.
Held-out perplexity — the same formula applied to a fixed test sentence that was never in the training data, the textbook way to score a language model on unseen text.

Each node in the 3D ring is a vocabulary word; arcs are the bigram transitions actually observed in training, brighter and thicker where the smoothed model assigns higher probability. The glowing walker samples a next word from the current node's distribution every tick and hops along the matching arc — literally the same random walk that drives greedy/temperature text generation in real language models.

Current word Start / End Vocabulary word

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.