🎯 3D Pass@k Unbiased Estimator: Hypergeometric Urn, Plug-in Bias and Benchmark Difficulty
A glass urn built in Blender holds n sampled programs, c of them correct; draw k without replacement and compare the unbiased pass@k estimate 1 - C(n-c,k)/C(n,k) with the naive plug-in value and the Beta-mixture truth, checked by exact binomial expectation, urn shuffles and numerical integration.
⚙ Under the hood
A glass urn built in Blender holds n sampled programs, c of them correct; draw k without replacement and compare the unbiased pass@k estimate 1 - C(n-c,k)/C(n,k) with the naive plug-in value and the Beta-mixture truth, checked by exact binomial expectation, urn shuffles and numerical integration.
pass@kCode Generation BenchmarksUnbiased EstimatorHypergeometric SamplingBeta DistributionEstimator BiasLLM EvaluationMonte Carlo
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install