Benchmark Contamination Detector

Ground Truth

Contamination rate 25%

Model's true skill 62%

Benchmark size 80 items

Decontamination Filter

Similarity threshold 80%

Accuracy

Reported (raw)

—

Decontaminated

—

Inflation

—

True skill

—

Detector Quality

Recall (leaks caught)

—

False-positive rate

—

Leaked items (ground truth)

—

Flagged as leaked (above threshold) Kept as clean (below threshold) Amber halo = truly memorised by the model

How it works

Each of the N glowing spheres is one benchmark question, arranged in a ring. Its height is the cosine similarity between that question and its nearest neighbour in the training corpus (the blue cloud at the centre) — a stand-in for real contamination checks that compare embeddings or n-gram overlap between eval sets and pretraining data.

reported accuracy  ≈ p_leak · p_correct|leak + (1 − p_leak) · true_skill
p_correct|leak      ≈ 0.97   (the answer was memorised, not reasoned)

Because leaked items get answered almost perfectly regardless of real ability, even a modest leak rate inflates the headline number. The translucent disc is the detection threshold: items above it are flagged and dropped before re-scoring, giving the "decontaminated accuracy" readout.

recall  = caught leaks / all true leaks   (raise threshold → recall falls)
FPR     = clean items wrongly flagged / all clean items   (raise threshold → FPR falls)

The trade-off is unavoidable: a low threshold catches more leakage but also discards genuinely novel questions (hurting sample size and precision); a high threshold is conservative but lets subtler leaks slip back into the "clean" set. This is exactly the judgment call behind benchmark-decontamination reports published alongside frontier model releases.

Training corpus (embedding cloud) Threshold plane

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.