🎯 3D Best-of-N Reward Hacking: Proxy vs Gold Reward, KL Budget and the Goodhart Peak
A Blender-built stage where candidate responses sit by proxy reward and true quality: pick the best of n by a flawed reward model, watch the gold reward rise then collapse as rare exploit responses win, and compare with the KL-regularised optimum, verified by order statistics, the KL formula and Monte Carlo.
⚙ Under the hood
A Blender-built stage where candidate responses sit by proxy reward and true quality: pick the best of n by a flawed reward model, watch the gold reward rise then collapse as rare exploit responses win, and compare with the KL-regularised optimum, verified by order statistics, the KL formula and Monte Carlo.
Best-of-NReward ModelGoodhart LawReward HackingKL DivergenceRLHFOver-optimisationAI Alignment
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install