Pass@k Benchmark Estimator: Sampling Grid & Score Curve (2D)

Benchmark Parameters

Tasks 12

Samples per task (n) 8

k (pass@k) 3

True success rate p 35%

Leaked tasks (contaminated) 0%

Live Score

pass@k (all tasks)

—

pass@k (clean only)

—

Leakage inflation

—

Seed-to-seed σ

—

correct sample cell

failed sample cell

leaked / contaminated task tag

pass@k(k) curve — all tasks

pass@k(k) curve — clean only

How it works

An agent attempts each task n times independently (different seeds / temperature). If c of those attempts succeed, the unbiased estimator for the probability that at least one of k randomly-drawn attempts (k ≤ n) succeeds is:

pass@k = 1 - C(n-c, k) / C(n, k)
       = 1 - ∏_{i=0}^{k-1} (n-c-i) / (n-i)

The left panel is a real sampling grid: one row per task, one cell per attempt, colored green for success and red for failure — the raw data the estimator is built from, not a stylised abstraction. The bar at the end of each row is that task's own pass@k value at the current k. The right panel plots the closed-form curve pass@k(k) for every k from 1 to n using the exact same per-task counts, so you can see the whole monotonic rise in one view instead of reading a single number — a higher k means more of the n attempts count toward "at least one success", so the curve only ever climbs as k grows — the vertical marker tracks the k slider live.

Tasks / Samples per task / k — shape of the benchmark run; the curve panel's x-axis spans 1..n so a larger n shows more of the estimator's shape at once.
True success rate p — the agent's real per-attempt success probability; each task's actual rate jitters slightly around p to mimic task difficulty variance.
Leaked tasks — the fraction of tasks whose ground-truth answer leaked into training data, so the agent scores near-perfectly on them regardless of real skill. Comparing the "all tasks" curve to the "clean only" curve shows exactly how much of a benchmark's headline number is contamination, not capability.
Seed-to-seed σ — the standard deviation of the pass@k(all) estimate across resamples with the same settings, showing why a single run's score is not a reliable read without repeated evaluation.

Sampling grid — one row per task, one cell per attemptpass@k(k) curve, k = 1..n