Traffic Signal Optimization: Adaptive Control

Fixed timing versus learning from the queue

A conventional traffic light runs a fixed-time cycle: north-south green for 30 seconds, then all-red clearance, then east-west green for 30 seconds, repeat forever, regardless of whether either approach has any cars on it at all. It is simple and predictable, and it wastes enormous amounts of green time on empty approaches at 2 a.m. and starves the busy approach at rush hour. This simulation instead trains a Q-learning agent live, in your browser, to control the same intersection by reading queue lengths and choosing which direction gets the green — and races it against the fixed-time baseline so the gap is visible, not just claimed.

Framing the intersection as a Markov Decision Process

Reinforcement learning needs a state, an action set, and a reward. Here the state is a discretised description of the intersection — typically the queue length on each approach, bucketed into a handful of levels, plus which phase is currently green and how long it has been green. The action is small: keep the current phase, or switch to the next one. The reward given after each action is negative, proportional to something the city actually cares about — total vehicles waiting, or cumulative wait time — so the agent is explicitly trying to minimise accumulated delay, not maximise throughput as an abstract number.

state  = (queue_N, queue_S, queue_E, queue_W, current_phase, phase_duration)
action = { keep_phase, switch_phase }
reward = - (total vehicles waiting this step)

Q-learning: the update rule

Q-learning maintains a table Q(s, a) estimating the long-run discounted return of taking action a in state s and then acting optimally forever after. Every step it observes a transition (s, a, r, s') and nudges the table toward the observed evidence:

Q(s, a) <- Q(s, a) + alpha * [ r + gamma * max_a' Q(s', a') - Q(s, a) ]

alpha = learning rate   (how much to trust the newest sample)
gamma = discount factor  (how much future reward matters vs. immediate)

The term in brackets is the temporal-difference error: the gap between what the agent expected (the old Q value) and what it now believes is closer to true (the reward just received plus the best estimate of everything after it). Repeated over thousands of intersection cycles, the table converges toward the true optimal action-value function, without ever being told the underlying queueing dynamics — it learns purely from trial, error and the reward signal.

Exploration versus exploitation

An agent that always picks the action with the highest current Q-value can get stuck exploiting a mediocre policy it found early, never discovering a better one. Epsilon-greedy exploration fixes this with a simple compromise: with probability epsilon take a random action instead of the greedy one, and decay epsilon over training so the agent explores heavily early (when its Q-table is still unreliable) and exploits confidently late (once the table has converged). This decay schedule is the single most visible knob in how fast the training curve in the demo climbs — too little exploration and the agent commits early to a poor policy; too much and it wastes cycles never settling.

action = random(actions)         with probability epsilon(t)
       = argmax_a Q(state, a)     with probability 1 - epsilon(t)
epsilon(t) decays from ~1.0 toward ~0.05 over training

Why the fixed-time baseline loses

Fixed timing is optimal only for the exact traffic pattern it was tuned for, and real traffic is not that pattern — it fluctuates by time of day, day of week, and unpredictable events. The Q-learning agent's queue-length state gives it something fixed timing structurally cannot have: it sees right now how much traffic is actually waiting on each approach, and it can hold a green a few extra seconds for a backed-up approach or cut a green short for an empty one. Over an episode with any real variance in arrival rates, that reactive advantage compounds into a materially lower average wait time — which is exactly the gap the two live-training curves in the demo make visible rather than asserted.

The trade-offs are real ones, not implementation details to wave away. A full state-action table scales poorly — every additional approach or every finer queue bucket multiplies the table size — which is why city-scale deployments increasingly use function approximation (Deep Q-Networks) instead of a raw table once the state space grows past a few intersections. And a Q-learning agent trained on one intersection's traffic distribution does not automatically generalise to a different intersection's geometry or arrival pattern; it has to be retrained or at minimum fine-tuned.

Frequently asked questions

Try it live

Everything above runs in your browser — open Traffic Signal Optimization and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.