This simulation turns LLM benchmarking into something you can watch happen: six model bars rise and fall across four benchmark columns — MMLU, HumanEval, BIG-Bench and Chatbot Arena — as sampling noise and benchmark weighting change the leaderboard in real time.
Each bar's observed score is the model's underlying skill plus random noise drawn from a normal distribution with standard error SE = σ/√n. Small benchmarks like HumanEval (only 164 problems) have a much noisier score than large ones like MMLU (about 14,000 questions) — the white whisker on top of each bar shows the resulting 95% confidence interval, and it visibly shrinks as you raise the sample size.
Drag Sample size n to see how more test items narrow the confidence interval, Task noise σ to make individual questions harder to grade consistently, and Weight: code ↔ knowledge to shift the composite leaderboard between knowledge/reasoning benchmarks and code/human-preference benchmarks. Press Reroll sample to redraw a fresh evaluation run, or Reset to return to the defaults.
Real leaderboards are sensitive to exactly this kind of weighting: a model can rank #1 on a code-heavy composite and drop several places on a knowledge-heavy one, which is why serious evaluation suites like HELM report many benchmarks side by side instead of collapsing everything into one score.
Benchmarking a language model means testing it on a fixed set of items — questions, coding problems, human comparisons — and averaging the result. That average is only an estimate of the model's true skill, and how noisy it is depends directly on how many items were used.
SE = σ / √n — standard error of the mean shrinks with more test items n (the Central Limit Theorem in action).
observed = true_skill + N(0, SE) — the score you actually measure is the true skill plus sampling noise.
composite = w · knowledge + (1 − w) · code — the leaderboard score is a weighted blend, and changing w can reorder it.
HumanEval has only 164 hand-written coding problems, while MMLU has around 14,000 questions — so two models with identical true coding skill can show a much wider score spread on HumanEval simply because n is smaller, exactly what the shrinking whisker in this simulation illustrates.