Reward hacking: a robot on the proxy surface (3D)

A Blender-built lab table: the true reward is the solid terrain, the proxy the red glass above it. The robot climbs the glass and may be lured to the decoy beacon.

Switch to 2D

Presets

Playback

Training

Camera

Layers

Quality

Model check

The science

Goodhart's law in one picture

"When a measure becomes a target, it ceases to be a good measure." An agent can only climb the reward it is given. Here the reward it is given, the proxy, is the true objective plus an extra narrow bump at a decoy that a designer did not intend:

R_true(p)  = G(p; goal,  s = 1.1)
R_proxy(p) = R_true(p) + k G(p; decoy, s = 0.6)
G(p; c, s) = exp( -|p - c|^2 / (2 s^2) )

How the agents learn

Every agent follows the proxy gradient with exploration noise (an Euler-Maruyama step):

dp = lr c0 grad R_proxy(p) dt + sigma sqrt(dt) N(0, I),   c0 = 2.2
grad G = -(p - c) / s^2 * G

The solid terrain in 3D is R_true and the red glass is R_proxy; the robot climbs the glass. At the decoy the proxy reads k while the true reward is about e^-19.8, so the Goodhart gap there equals k.

What to look for

  • The decoy peak is narrow (s = 0.6) and the goal peak is wide (s = 1.1). Many agents still reach the goal, but every agent that starts inside the decoy's basin is captured. Raising k deepens the pull and mostly raises the gap; it moves the basin edge only a little (see the sweep graph).
  • Noise lets an agent leave one basin and fall into another; with a very small learning rate agents hardly move at all.
  • In reward space (2D page) an aligned agent lies on the diagonal. A hacked agent sits high above it: it scores well on the proxy and almost nothing on the real goal.

Honest simplifications

This is a toy landscape, not a model of a real reinforcement-learning system: two Gaussian bumps, plain gradient ascent and Gaussian noise. Real reward hacking involves environments, policies and learned reward models. The checks prove the code does what these equations say (gradient, fixed points, noise spread, monotone effect of k); they do not prove anything about real AI systems.

Loading model...
Loading Blender model...
Share of agents aligned, hacked and still searching
Focus agent: proxy reward, true reward and the Goodhart gap
Noiseless agents ending on the decoy against decoy strength k
Made with 🟠 Blender · Three.js