This is the same Goodhart's Law dynamics as the 3D version of this simulator, but plotted in a genuinely different space. Instead of watching an agent's position on a hill, every agent is placed as one point at coordinates (R_true, R_proxy) — its true reward on the x-axis, the proxy reward it actually optimizes on the y-axis:
R_true(x,z) = G(x,z; goal, σ=1.1)
R_proxy(x,z) = R_true(x,z) + k·G(x,z; decoy, σ=0.6)
G(p; c, σ) = exp( -|p-c|² / (2σ²) )
The hidden (x,z) state still evolves by numerical-gradient ascent on the proxy surface plus exploration noise — identical update rule to the 3D version:
v = lr · ∇R_proxy(x,z) + noise·𝒩(0,1)
(x,z) += v · dt
but nothing about (x,z) is drawn. What you see is a reward-space phase portrait: the diagonal line is R_proxy = R_true, i.e. perfect alignment — a proxy that never diverges from the truth. Any point above the diagonal is being over-rewarded relative to its real merit; the shaded band is the "reward-hacking zone". The green marker is where the true-goal attractor sits in this space (≈(1, 1)); the red marker is where the decoy attractor sits (≈(0, k)) — as k grows, the decoy marker climbs the y-axis even though its x-coordinate (true merit) stays at zero. A swarm of 140 independently-seeded agents is plotted at once so you can read off the population statistics directly, and the strip below tracks what fraction of the swarm is aligned vs. hacked over training time.
- Decoy strength k — how much taller the misleading proxy peak is than the true goal in the reward the agent actually optimizes; it sets the decoy marker's height in reward-space.
- Learning rate — how aggressively each agent's policy follows the proxy gradient each step.
- Exploration noise — stochastic policy noise; it's also what lets an agent random-walk out of the true goal's broader basin and into the decoy's narrower one.
- Alignment gap — proxy reward minus true reward for the focus agent; the vertical distance of its point above the diagonal.
Numerically verifying this build: the true-goal basin (σ=1.1) is wider than the decoy's (σ=0.6), so most randomly-seeded agents converge to true-goal regardless of k — only a noise-driven excursion into the decoy's narrower basin produces a hacked outcome. That basin-size effect, not just which peak is taller, is what the population panel is actually measuring.