Presets
Playback
Training
Camera
Layers
Quality
Model check
The science
Goodhart's law in one picture
"When a measure becomes a target, it ceases to be a good measure." An agent can only climb the reward it is given. Here the reward it is given, the proxy, is the true objective plus an extra narrow bump at a decoy that a designer did not intend:
R_true(p) = G(p; goal, s = 1.1) R_proxy(p) = R_true(p) + k G(p; decoy, s = 0.6) G(p; c, s) = exp( -|p - c|^2 / (2 s^2) )
How the agents learn
Every agent follows the proxy gradient with exploration noise (an Euler-Maruyama step):
dp = lr c0 grad R_proxy(p) dt + sigma sqrt(dt) N(0, I), c0 = 2.2 grad G = -(p - c) / s^2 * G
The solid terrain in 3D is R_true and the red glass is R_proxy; the robot climbs the glass. At the decoy the proxy reads k while the true reward is about e^-19.8, so the Goodhart gap there equals k.
What to look for
- The decoy peak is narrow (s = 0.6) and the goal peak is wide (s = 1.1). Many agents still reach the goal, but every agent that starts inside the decoy's basin is captured. Raising
kdeepens the pull and mostly raises the gap; it moves the basin edge only a little (see the sweep graph). - Noise lets an agent leave one basin and fall into another; with a very small learning rate agents hardly move at all.
- In reward space (2D page) an aligned agent lies on the diagonal. A hacked agent sits high above it: it scores well on the proxy and almost nothing on the real goal.
Honest simplifications
This is a toy landscape, not a model of a real reinforcement-learning system: two Gaussian bumps, plain gradient ascent and Gaussian noise. Real reward hacking involves environments, policies and learned reward models. The checks prove the code does what these equations say (gradient, fixed points, noise spread, monotone effect of k); they do not prove anything about real AI systems.