🎯 2D Best-of-N Reward Hacking: Proxy vs Gold Reward, KL Budget and the Goodhart Peak
Scatter view of best-of-n selection against an imperfect proxy reward: exact gold and proxy curves versus n and KL, the Goodhart peak caused by rare exploit responses, and the KL-regularised tilt, verified by closed-form order statistics, direct integration of the KL and Monte Carlo selection runs.
⚙ Under the hood
Scatter view of best-of-n selection against an imperfect proxy reward: exact gold and proxy curves versus n and KL, the Goodhart peak caused by rare exploit responses, and the KL-regularised tilt, verified by closed-form order statistics, direct integration of the KL and Monte Carlo selection runs.
Best-of-NReward ModelGoodhart LawReward HackingKL DivergenceRLHFOver-optimisationAI Alignment
2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install