Maximization Bias: Q-Learning vs Double Q-Learning

This simulator runs two reinforcement-learning agents side by side on the textbook environment for demonstrating maximization bias: a start state with a cheap, certain "right" action and a tempting "left" action that leads into a bank of noisy-reward arms with an equal-or-worse true mean. Standard Q-learning's max-over-noisy-estimates bootstrap makes it overvalue the noisy branch and drift toward taking "left" far more than the ~5% that pure exploration accounts for; Double Q-learning, which decouples action selection from action evaluation across two independent tables, converges back down near the true optimum. A live 3D bar graph shows either agent's current per-arm value estimates growing and shrinking around state B, two rolling curves track each agent's left-action rate over thousands of episodes, and a live bias readout reports exactly how far Q-learning's maximum estimate has drifted from the true mean reward.