🎯 2D Pass@k Unbiased Estimator: Hypergeometric Urn, Plug-in Bias and Benchmark Difficulty
Benchmark grid of problems by samples with the pass@k curve: the unbiased estimator versus the biased plug-in formula, bias as a function of n, and the Beta difficulty model, verified by a log-gamma form, exact binomial expectation, 200 000 urn shuffles and midpoint integration.
⚙ Under the hood
Benchmark grid of problems by samples with the pass@k curve: the unbiased estimator versus the biased plug-in formula, bias as a function of n, and the Beta difficulty model, verified by a log-gamma form, exact binomial expectation, 200 000 urn shuffles and midpoint integration.
pass@kCode Generation BenchmarksUnbiased EstimatorHypergeometric SamplingBeta DistributionEstimator BiasLLM EvaluationMonte Carlo
2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install