🚦 Traffic Signal Optimization: Adaptive Control

Q(s,a) ← Q(s,a) + α[r + γ·max_a′ Q(s′,a′) − Q(s,a)]. After taking action a in state s and observing reward r and next state s′, the agent moves its estimate of the action's value Q(s,a) toward the observed reward plus the discounted value of the best action available in the next state. α is the learning rate (how much each observation updates the table) and γ is the discount factor weighting future reward against immediate reward.

With probability ε the agent picks a uniformly random phase (exploration), and with probability 1−ε it picks the phase with the highest current Q-value for the current state (exploitation). Pure exploitation from the start would let the agent get stuck on a mediocre policy it discovered early; pure exploration would never converge to a good policy at all. Epsilon-greedy is the simplest practical compromise, and the ε slider in this simulation lets you feel that trade-off directly — high ε makes the agent visibly more erratic, low ε makes it settle into whatever policy it already knows.

Yes, subject to real caveats. Q-learning is proven to converge to the optimal action-value function for a Markov decision process, provided every state-action pair is visited infinitely often (which requires ε to never fully vanish) and the learning rate is scheduled appropriately. In this simplified, finite training run you will see the average wait time per episode trend downward with visible noise — noise that never fully disappears because traffic arrivals are stochastic and the state space is small enough that individual unlucky episodes still happen even under a near-optimal policy.

SCOOT (Split, Cycle and Offset Optimisation Technique, developed in the UK) and SCATS (Sydney Coordinated Adaptive Traffic System, developed in Australia) are real, widely deployed adaptive traffic-signal systems used in hundreds of cities worldwide. Both use loop detectors or cameras to measure real-time traffic flow and continuously adjust signal splits, cycle lengths and offsets between intersections, rather than relying on a fixed schedule. They are the production-grade descendants of the same core idea demonstrated here in miniature: let measured queue and flow data drive the signal timing instead of a preset clock.

Tabular Q-learning needs a finite state space — a separate table row for every distinct state. Using the exact queue count (0, 1, 2, 3, ... up to dozens of cars) on four approaches would blow up the table size and require far more experience to fill in every entry with a useful estimate. Bucketing each approach's queue into a handful of levels (empty / light / moderate / heavy) keeps the table small enough to learn quickly while still capturing the traffic information that matters for the phase-choice decision.

The reward for each decision window is the negative of total queue-seconds accumulated across all approaches during that window — a direct proxy for delay. Minimizing the sum of negative rewards (equivalently, maximizing cumulative reward) is mathematically equivalent to minimizing total vehicle-seconds of delay, which is exactly what the "average wait" readout tracks. This is the same quantity traffic engineers use to justify adaptive-signal deployments in the real world.

This is the same reinforcement-learning machinery used in the grid-world Reinforcement Learning simulation, applied to a continuous, stochastic queueing domain instead of a discrete grid. It also connects to the Self-Driving Car and Robot Arm Pick and Place simulations, which use different learning and control approaches to solve other real-time decision problems for autonomous systems.