Each glowing dot is one benchmark question, arranged around a ring — drag the ring to spin it. Its radius from centre is the cosine similarity between that question and its nearest neighbour in the training corpus — a stand-in for real contamination checks that compare embeddings or n-gram overlap between eval sets and pretraining data. The lower panel is a histogram of that same similarity value across all items.
reported accuracy ≈ p_leak · p_correct|leak + (1 − p_leak) · true_skill
p_correct|leak ≈ 0.97 (the answer was memorised, not reasoned)
Because leaked items get answered almost perfectly regardless of real ability, even a modest leak rate inflates the headline number. The dashed gold circle is the detection threshold: items outside it are flagged and dropped before re-scoring, giving the "decontaminated accuracy" readout.
recall = caught leaks / all true leaks (raise threshold → recall falls)
FPR = clean items wrongly flagged / all clean items (raise threshold → FPR falls)
The trade-off is unavoidable: a low threshold catches more leakage but also discards genuinely novel questions (hurting sample size and precision); a high threshold is conservative but lets subtler leaks slip back into the "clean" set. This is exactly the judgment call behind benchmark-decontamination reports published alongside frontier model releases.
Honest simplifications. The items are synthetic: a leaked item gets a nearest-neighbour similarity drawn uniformly from 0.70–0.99 and is answered correctly with probability 0.97; a clean item gets similarity 0.72·u1.7 and is answered with the model's true skill. These shapes and the 0.97 value are illustrative parameters, not measured from a real model. Real audits use n-gram overlap or embedding search whose score distributions overlap far more and depend on paraphrase. Accuracy is a Bernoulli estimate, so small benchmarks (20–150 items) fluctuate — press Resample to see the spread. Closed forms used in the checks: raw accuracy = c·0.97 + (1−c)·skill; recall = (0.99−th)/0.29; FPR = 1−(th/0.72)1/1.7 for th<0.72.