Maximization Bias: Q-Learning vs Double Q-Learning

Environment

Learning rate α 0.10

Reward arms at state B 8

Mean reward at B -0.10

Simulation speed 200 ep/s

3D graph shows

Live Stats

Episodes

Optimal left%

5.0%

Left% · Q-learning

—

Left% · Double Q

—

Bias · max Q(B) − true mean

—

Q-learning left-rate (rolling)

Double Q left-rate (rolling)

Optimal ≈5% reference

How it works

This is the classic "maximization bias" MDP: from state A, action right ends the episode with reward 0; action left moves to state B, which has N actions, each returning a reward drawn from N(mean, 1) — by default with a mean below 0. Since every arm at B has the same (non-positive) true mean, the truly optimal policy always picks right, and an ε-greedy agent should pick left only ε/2 of the time (≈5%) from exploration alone.

Q-learning update:
Q(s,a) ← Q(s,a) + α[ r + γ·max Q(s',a') − Q(s,a) ]

Double Q-learning (updates Q1 or Q2 at random each step):
Q1(s,a) ← Q1(s,a) + α[ r + γ·Q2(s', argmax Q1(s',a')) − Q1(s,a) ]

Because Q-learning's bootstrap target uses max over its own noisy estimates, and E[max Q̂] ≥ max E[Q̂] for noisy Q̂, the agent's estimate of state B is biased upward early in training — it looks better than it truly is, so the agent visits left far more often than the optimal 5%. Double Q-learning breaks this by selecting the best action with one table and evaluating it with an independent second table, removing the correlation that causes the bias.

Learning rate α — step size of every update; higher α reacts faster but stays noisier.
Reward arms at B — more arms means more chances for a lucky noisy sample to look like a winner, amplifying the bias.
Mean reward at B — how costly the "left" mistake truly is; the more negative, the more obvious the bias is once it corrects.
3D graph toggle — switches which estimator's current value estimates drive the bar heights around state B in the 3D scene.

Real-world relevance: this exact bias motivated Double DQN, the fix used in almost every modern deep Q-learning system (Atari, robotics control) to stop value networks from silently drifting toward overconfident, wrong actions.

State A Terminal (right, r=0) State B

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.