Home▸AI & Machine Learning▸3D Best-of-N Reward Hacking: Proxy vs Gold Reward, KL Budget and the Goodhart Peak

🎯 3D Best-of-N Reward Hacking: Proxy vs Gold Reward, KL Budget and the Goodhart Peak

A Blender-built stage where candidate responses sit by proxy reward and true quality: pick the best of n by a flawed reward model, watch the gold reward rise then collapse as rare exploit responses win, and compare with the KL-regularised optimum, verified by order statistics, the KL formula and Monte Carlo.

AI & Machine Learning3DModerate60 FPS📱 Mobile-adapted🟠 Blender⇄ 2D version
3D Best-of-N Reward Hacking: Proxy vs Gold Reward, KL Budget and the Goodhart Peak ↗ Open standalone
⚙ Under the hood

A Blender-built stage where candidate responses sit by proxy reward and true quality: pick the best of n by a flawed reward model, watch the gold reward rise then collapse as rare exploit responses win, and compare with the KL-regularised optimum, verified by order statistics, the KL formula and Monte Carlo.

Best-of-NReward ModelGoodhart LawReward HackingKL DivergenceRLHFOver-optimisationAI Alignment

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)