Home▸AI & Machine Learning▸Reward Hacking in Reward-Space: A Goodhart's Law Phase Portrait

Reward Hacking in Reward-Space: A Goodhart's Law Phase Portrait

A 2D phase-space plot of Goodhart's Law: instead of watching an agent climb a 3D hill, plot every agent's (true reward, proxy reward) pair directly. The diagonal is perfect alignment; the further a swarm drifts above it, the harder the proxy is being gamed.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

The 3D version of this simulator renders a literal reward landscape and lets you watch a single agent climb it. This 2D companion runs the identical hidden dynamics — the same Gaussian true/proxy reward fields, the same finite-difference gradient ascent plus exploration noise — but never draws a landscape at all. Instead every agent (and a full population of 140 at once) is plotted as a single point in reward space: true reward on one axis, proxy reward on the other. Perfect alignment is the diagonal; Goodhart's Law is the swarm drifting above it. A scrolling strip beneath tracks the population's aligned-vs-hacked split over training time, turning a single anecdote into a statistic.

⚙ Under the hood

A 2D phase-space plot of Goodhart's Law: instead of watching an agent climb a 3D reward hill, plot every agent's (true reward, proxy reward) pair directly. The diagonal line is perfect alignment; a population of 140 agents drifting above it into the shaded zone is Goodhart's Law happening live, with a scrolling strip tracking the aligned-vs-hacked split over training time.

ai safetyreward hackingalignmentreinforcement learninggoodhart's lawspecification gamingphase space

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install