🚦 Traffic Signal Optimization: Adaptive Control
A real Q-learning agent watches queue lengths at a four-way intersection and learns, through genuine trial-and-error, which phase to run next — compare it live against a fixed-time baseline that ignores traffic entirely.
Drag to orbit · Scroll to zoom · Top-down intersection view
Learning curve — average wait time per episode
Fixed-Time Control
Each phase runs for a preset duration regardless of how many cars are actually waiting. Switch to Adaptive Q-Learning and try the Asymmetric Preset to see why fixed timing wastes green time on empty approaches.
Fixed-Time vs. Adaptive Q-Learning Control
Fixed-time control — Each phase (N-S green,
E-W green) gets a preset duration, cycling forever regardless of queue state. Simple,
cheap, and used on millions of low-traffic intersections worldwide — but provably
suboptimal whenever traffic is variable or asymmetric, because green time is wasted on
empty approaches while cars queue on busy ones.
Adaptive Q-learning control — A tabular
reinforcement-learning agent observes the discretized queue length on every approach
(its state), picks which phase to run next (its action), and receives
a reward equal to the negative total queue length accumulated during that decision
window. It updates a real Q-table with the Bellman update rule
Q(s,a) ← Q(s,a) + α[r + γ·max Q(s′,a′) − Q(s,a)] after every decision, using
an epsilon-greedy policy to balance exploring new phase choices against exploiting the
best one found so far. Given enough exploration, this update provably converges toward
the optimal action-value function for the underlying Markov decision process.
About Traffic Signal Optimization
This simulation renders a four-way intersection in real 3D and lets you compare two genuinely different control strategies. Fixed-time control cycles through North-South and East-West phases on a preset schedule, exactly like the simplest traffic controllers deployed since the 1920s. Adaptive control is a real tabular Q-learning agent: it discretizes the queue length on each approach into a state, chooses a phase as its action, and updates a Q-table live from the reward it observes — negative total queue length — using the standard Bellman update rule. Nothing about the adaptive behaviour is scripted; watch the learning curve and you will see genuine noisy improvement, not a canned downward line.
Deployed adaptive systems such as SCOOT (UK) and SCATS (Australia) use similar queue- and occupancy-based feedback loops to retime signals in real cities, typically cutting average delay by 10-25% versus fixed-time plans, especially under asymmetric or time-varying demand.
Frequently Asked Questions
What state, action and reward does the Q-learning agent use?
The state is the queue length on each of the four approaches, discretized into buckets (0, 1-2, 3-5, 6+ cars) and combined into a single state index. The action is which of the two phases to run next — North-South green or East-West green. The reward for a decision window is the negative of the total queue length accumulated across all approaches during that window, so the agent is directly rewarded for keeping queues short.
How do I use this simulation?
Pick Fixed-Time or Adaptive Q-Learning with the mode buttons. Adjust the four arrival-rate sliders per approach, or click the Asymmetric Preset to send far more traffic north-south than east-west. In Fixed-Time mode, tune the preset green durations. In Adaptive mode, tune the learning rate α and exploration rate ε, then watch the learning curve as episodes accumulate — Reset/Retrain clears the Q-table and starts over.
Why is fixed-time control provably suboptimal under asymmetric traffic?
A fixed-time controller allocates green time by a schedule set in advance, with no information about which approach actually has cars waiting. Under asymmetric or time-varying demand, this guarantees that green time is sometimes given to an empty approach while cars queue on a busy one — a structurally suboptimal allocation for any traffic pattern that differs from the one the schedule was tuned for. An adaptive controller that can observe the real queue state can never do worse, and typically does substantially better, because it can always choose to replicate the fixed-time policy but is not restricted to it.
What is the Q-learning update rule shown in this simulation?
Q(s,a) ← Q(s,a) + α[r + γ·max_a′ Q(s′,a′) − Q(s,a)]. After taking action a in state s and observing reward r and next state s′, the agent moves its estimate of the action's value Q(s,a) toward the observed reward plus the discounted value of the best action available in the next state. α is the learning rate (how much each observation updates the table) and γ is the discount factor weighting future reward against immediate reward.
What does epsilon-greedy exploration mean here?
With probability ε the agent picks a uniformly random phase (exploration), and with probability 1−ε it picks the phase with the highest current Q-value for the current state (exploitation). Pure exploitation from the start would let the agent get stuck on a mediocre policy it discovered early; pure exploration would never converge to a good policy at all. Epsilon-greedy is the simplest practical compromise, and the ε slider in this simulation lets you feel that trade-off directly — high ε makes the agent visibly more erratic, low ε makes it settle into whatever policy it already knows.
Does the Q-learning agent actually converge to a good policy?
Yes, subject to real caveats. Q-learning is proven to converge to the optimal action-value function for a Markov decision process, provided every state-action pair is visited infinitely often (which requires ε to never fully vanish) and the learning rate is scheduled appropriately. In this simplified, finite training run you will see the average wait time per episode trend downward with visible noise — noise that never fully disappears because traffic arrivals are stochastic and the state space is small enough that individual unlucky episodes still happen even under a near-optimal policy.
What are SCOOT and SCATS?
SCOOT (Split, Cycle and Offset Optimisation Technique, developed in the UK) and SCATS (Sydney Coordinated Adaptive Traffic System, developed in Australia) are real, widely deployed adaptive traffic-signal systems used in hundreds of cities worldwide. Both use loop detectors or cameras to measure real-time traffic flow and continuously adjust signal splits, cycle lengths and offsets between intersections, rather than relying on a fixed schedule. They are the production-grade descendants of the same core idea demonstrated here in miniature: let measured queue and flow data drive the signal timing instead of a preset clock.
Why use discretized queue-length buckets instead of exact counts?
Tabular Q-learning needs a finite state space — a separate table row for every distinct state. Using the exact queue count (0, 1, 2, 3, ... up to dozens of cars) on four approaches would blow up the table size and require far more experience to fill in every entry with a useful estimate. Bucketing each approach's queue into a handful of levels (empty / light / moderate / heavy) keeps the table small enough to learn quickly while still capturing the traffic information that matters for the phase-choice decision.
How does the reward relate to real average wait time?
The reward for each decision window is the negative of total queue-seconds accumulated across all approaches during that window — a direct proxy for delay. Minimizing the sum of negative rewards (equivalently, maximizing cumulative reward) is mathematically equivalent to minimizing total vehicle-seconds of delay, which is exactly what the "average wait" readout tracks. This is the same quantity traffic engineers use to justify adaptive-signal deployments in the real world.
How does this relate to other simulations on this site?
This is the same reinforcement-learning machinery used in the grid-world Reinforcement Learning simulation, applied to a continuous, stochastic queueing domain instead of a discrete grid. It also connects to the Self-Driving Car and Robot Arm Pick and Place simulations, which use different learning and control approaches to solve other real-time decision problems for autonomous systems.