Deep Q-Network: Learning to Play from Pixels

The Core Idea: Learning the Value of Actions

Q-learning is built around an action-value function, Q(s, a), which estimates the total future reward an agent can expect if it takes action a in state s and then behaves optimally afterward. Classically, this function was stored as a giant lookup table: one entry per state-action pair, updated bit by bit as the agent explored. That works fine for small, discrete worlds like a grid maze, but it collapses completely for something like an Atari screen, where the raw pixel input defines an astronomically large space of possible states. There simply aren't enough table cells, or enough experience, to visit every state even once.

Swapping the Table for a Neural Network

DQN's central trick is to replace the impossible lookup table with a deep neural network that takes raw pixel frames as input and outputs an estimated Q-value for every possible action. Because the network generalises, it can produce sensible value estimates for states it has never seen exactly before, based on similarity to states it has. In the original DeepMind paper, this was a convolutional neural network trained on stacks of recent game frames, letting it implicitly infer motion and game dynamics purely from vision, exactly as a human player would.

Two Tricks That Made It Actually Work

Training a neural network on raw reinforcement learning data is notoriously unstable, so DQN introduced two key stabilisers. Experience replay stores past transitions (state, action, reward, next state) in a memory buffer and trains on randomly sampled batches from it, breaking the strong correlations between consecutive frames that would otherwise destabilise learning. A separate, slowly-updated target network supplies the Q-value targets used in training, so the goalposts the main network is chasing don't shift on every single update. Together, these two ideas turned an unstable idea into a reliably trainable algorithm.

From Breakout to the Deep RL Boom

DQN's 2015 Nature paper, Human-level control through deep reinforcement learning, showed a single architecture and hyperparameter set learning 49 different Atari 2600 games from pixels alone, matching or beating human performance on the majority of them. It is widely credited with kickstarting the modern wave of deep reinforcement learning research, inspiring successors like Double DQN, Dueling DQN, and eventually systems like AlphaGo. DQN still has real limits, though: it is notoriously sample-inefficient, can be unstable or diverge during training, and its standard form only handles discrete action spaces, which is why later algorithms extended it to continuous control.

Frequently asked questions

Try it live

Everything above runs in your browser — open Deep Q-Network: Learning to Play from Pixels and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.