The coloured surface is a reward landscape over a 2D response-style space. The tall narrow peak is the genuinely aligned optimum; the smaller bump near the start is a reward-hacking trap โ a style that scores deceptively well early on. The grey sphere is the frozen reference model (the SFT checkpoint every other method is measured against); the glowing sphere is the policy being trained, leaving a trail of its own colour. The particle cloud around the policy is a batch of sampled candidate responses, coloured by reward; the bright green/red pair is the chosen/rejected sample actually used this step.
L = โlog ฯ_ฮธ(y|x) โ imitate a fixed gold response directly, ignoring the reward landscape.max E[r(x,y)] โ ฮฒยทKL(ฯ_ฮธโฯ_ref) โ climb the reward gradient while a KL penalty pulls back toward the reference model.L = โlog ฯ(ฮฒ log(ฯ_ฮธ(y_w)/ฯ_ref(y_w)) โ ฮฒ log(ฯ_ฮธ(y_l)/ฯ_ref(y_l))) โ the closed-form solution of the RLHF objective, trained directly on (chosen, rejected) pairs with no separate reward model.L = L_SFT + ฮปยทL_OR, where L_OR = โlog ฯ(log(odds_ฮธ(y_w)/odds_ฮธ(y_l))) โ single-stage, reference-free: no KL anchor, just an odds-ratio penalty against the rejected style.r(x,y) comes from an AI judge instead of a human rater โ cheaper to scale, noisier signal.DPO (Rafailov et al., 2023) proved that the RLHF objective's optimal policy has a closed form, which is why DPO can skip training a reward model and running PPO rollouts entirely โ one classification-style loss on preference pairs does the same job. ORPO (Hong et al., 2024) went further and dropped the reference model too, folding preference and instruction-following into one stage.
This simulation drops a single policy onto a 3D reward landscape and lets you retrain it from scratch under five real alignment objectives โ SFT, RLHF/PPO, DPO, ORPO and RLAIF โ using the actual update rules from each method's paper, not a scripted animation. A grey sphere marks the frozen reference model; the glowing sphere is the policy, leaving a coloured trail as it trains. The tall peak is the genuinely aligned optimum, the smaller bump near the start is a reward-hacking trap, and the particle cloud shows sampled candidate responses coloured by reward, with the chosen/rejected pair highlighted for methods that train on preference pairs.
A live reward-landscape mesh with a reference-model marker, a trained policy marker and trail, and an instanced cloud of sampled candidate responses. Switching method resets the policy to the reference point and re-trains it live with that method's real objective.
Pick a method (SFT / RLHF / DPO / ORPO / RLAIF), then tune learning rate ฮฑ, KL coefficient ฮฒ (ignored by ORPO โ it's reference-free) and preference/AI-judge noise. Drag to orbit, scroll to zoom, and use Pause/Reset to freeze or restart training from the reference model.
DPO (Rafailov et al., 2023) proved the RLHF objective's optimal policy has a closed form, letting it skip the reward model and PPO rollouts entirely. ORPO (Hong et al., 2024) went further and dropped the reference model too, folding SFT and preference training into one stage.
SFT trains the model to imitate a fixed set of curated instruction/response pairs by minimising the negative log-likelihood of the gold response. It moves the policy straight toward the demonstrated target, regardless of the shape of the reward landscape around it.
RLHF trains a reward model on human preference data, then optimises the policy with reinforcement learning (typically PPO) to maximise expected reward while a KL penalty keeps it close to the reference (SFT) model. Unlike SFT, it can improve on responses it was never shown verbatim.
Direct Preference Optimization (DPO) shows that the RLHF objective has a closed-form optimal policy, letting you skip the separate reward model and PPO loop entirely: a single classification-style loss on preference pairs implicitly performs the same KL-constrained reward maximisation.
Odds Ratio Preference Optimization (ORPO) combines the SFT loss with an odds-ratio penalty in a single stage, with no reference model and no separate SFT warm-up phase. It is cheaper to train but relies on the odds-ratio term alone to keep rejected-style outputs suppressed.
Reinforcement Learning from AI Feedback (RLAIF) is RLHF with the human-labelled reward replaced by an AI judge model. It scales far more cheaply than human labelling but the reward signal is noisier and can drift from what a human would actually prefer.
Reward hacking happens when a policy finds a response style that scores well on the reward signal without actually being better. The smaller bump near the starting point is exactly that kind of trap โ methods with a weaker pull back toward the reference model are more exposed to it.