Maximization Bias 2D: Q-Learning vs Double Q-Learning

This simulator runs two reinforcement-learning agents side by side on the textbook environment for demonstrating maximization bias: a start state with a cheap, certain "right" action and a tempting "left" action that leads into a bank of noisy-reward arms with an equal-or-worse true mean. Standard Q-learning's max-over-noisy-estimates bootstrap makes it overvalue the noisy branch and drift toward taking "left" far more than the ~5% that pure exploration accounts for; Double Q-learning, which decouples action selection from action evaluation across two independent tables, converges back down near the true optimum. A draggable 2D radial graph shows either agent's current per-arm value estimates as bars growing and shrinking around state B, a rolling line chart tracks each agent's left-action rate over thousands of episodes against a 5% reference line, and a live bias readout reports exactly how far Q-learning's maximum estimate has drifted from the true mean reward.