Q-learning in continuous state spaces: from lookup tables to Deep Q-Networks

Why tabular Q-learning breaks

The classic Q-learning update refines a table Q(s, a) toward the observed reward plus the best achievable future value:

This requires visiting every (s, a) pair many times to get an accurate estimate. When the state is continuous — position, velocity, joint angle — there are infinitely many states, so no state is ever visited twice, and a table can never converge. Worse, a table gives you zero generalization: learning about state (1.001, 0.5) tells you nothing about the nearly identical state (1.002, 0.5).

Tile coding: discretizing space cheaply

A classic, cheap fix is tile coding: overlay several offset grids ("tilings") on the continuous state space. Each tiling activates exactly one tile per state, and the state's feature vector is the concatenation of one-hot tile indicators across all tilings. Nearby states share most of their active tiles, which gives coarse but effective generalization for free.

Linear function approximation

With a feature vector φ(s, a) from tile coding (or any hand engineered basis), Q is approximated as a linear combination of weights:

This is exactly the same TD-error rule as tabular Q-learning, just applied to weights instead of table cells — and it was the state-of-the-art for continuous control (with tile coding, RBF, or Fourier bases) for decades before deep learning.

Deep Q-Networks: a neural Q-function

Deep Q-Networks (Mnih et al., 2015) replace the hand-designed feature vector with a neural network Q(s, a; θ) that learns its own features end to end from raw state input (pixels, joint angles, etc.). The TD-error becomes a squared-error loss minimized by gradient descent:

In principle this is "just" Q-learning with gradient descent instead of a per-weight update rule. In practice, plugging a neural network directly into the naive Q-learning loop diverges almost immediately — two engineering tricks were needed to make it stable.

Replay buffers and target networks

Without a replay buffer and target network, the Q-function chases its own tail: every gradient step changes the bootstrap target itself, and the whole system can oscillate or diverge instead of converging.

A minimal DQN update in JavaScript

function dqnUpdate(onlineNet, targetNet, batch, gamma = 0.99) {
  let loss = 0;
  for (const { s, a, r, sNext, done } of batch) {
    const qValues = onlineNet.forward(s);
    const qNextTarget = done ? 0 : Math.max(...targetNet.forward(sNext));
    const tdTarget = r + gamma * qNextTarget;
    const tdError = tdTarget - qValues[a];

    loss += tdError * tdError;
    onlineNet.backwardStep(s, a, tdError); // gradient step on Q(s,a) only
  }

  // Periodically hard-copy (or soft-average with Polyak update) online → target
  if (onlineNet.step % 1000 === 0) targetNet.copyFrom(onlineNet);
  return loss / batch.length;
}

Where continuous Q-learning is used

• Atari and video games: the original DQN benchmark — raw pixel frames as continuous, high-dimensional state.
• Robotic control: joint angles and velocities as continuous state, though continuous actions usually push toward actor-critic methods (DDPG, SAC) instead.
• Autonomous driving simulators: sensor readings and vehicle dynamics as continuous state for lane-keeping and collision-avoidance policies.
• Recommendation systems: user embeddings as continuous state, discrete item choice as action.
• Resource allocation / scheduling: continuous load or queue-length signals driving discrete allocation decisions.