Question 1

What is Q-learning?

Accepted Answer

Q-learning is a model-free reinforcement learning algorithm that learns the expected long-run value, Q(state, action), of taking each action in each state, without needing a model of the environment's dynamics. It updates a table of Q-values using the Bellman equation, Q(s,a) ← Q(s,a) + α[r + γ·max Q(s′,a′) − Q(s,a)], where α is the learning rate, γ the discount factor, and r the reward received after acting. Over many trials the Q-table converges toward the true optimal action-value function, from which a greedy policy (always pick the highest-value action) becomes near-optimal.

Question 2

How is the traffic intersection turned into a Markov decision process?

Accepted Answer

The continuous, real-valued traffic state is discretized: each of the four approach queues (North, South, East, West) is bucketed into five levels (0, 1, 2, 3, 4-or-more vehicles), and combined with the current signal phase (North-South green or East-West green) to form one of 1,250 possible states. The agent has just two actions available at each decision point: keep the current phase, or attempt to switch it (subject to a minimum green-time constraint so lights cannot flicker). This is exactly the state/action/reward structure a tabular Q-learning agent needs.

Question 3

What reward signal drives the learning?

Accepted Answer

After every decision step the simulator counts the total vehicle-seconds of delay accumulated across all four approaches during that step (an accepted proxy for real intersection wait time) and hands the agent a reward equal to the negative of that delay, with an extra penalty subtracted whenever a phase switch actually occurs (representing the lost time of an amber/all-red clearance interval). Because the agent is rewarded for minimising delay, its learned policy is literally the timing strategy that reduces simulated wait time the most, not a scripted or pre-baked schedule.

Question 4

Why does epsilon-greedy exploration matter?

Accepted Answer

Early in training the Q-table is all zeros, so a purely greedy agent would get stuck repeating whatever action it tried first. Epsilon-greedy action selection picks a random action with probability epsilon and the current best-known action otherwise, guaranteeing every state-action pair eventually gets sampled so its Q-value can be learned. Epsilon is decayed each episode toward a small floor so the agent explores heavily at first and increasingly exploits its learned policy as training progresses, which is exactly the epsilon curve you can watch decay live in the stats panel.

Question 5

Is the learning curve genuine or scripted?

Accepted Answer

It is computed live from the agent's own performance: every 120 decision steps (one simulated episode) the simulator divides the total accumulated vehicle-seconds of delay by the number of vehicles that arrived, giving a true average wait time for that episode, and plots it as a new point. No target curve is drawn in — if you set the hyperparameters badly (for example alpha too high or epsilon decaying too fast) the curve can plateau early or stay noisy, exactly as it would for a real tabular Q-learning agent.

🚦 Optymalizator sygnalizacji świetlnej — Q-learning na żywo

O tej symulacji

🔬 Co pokazuje

🎮 Jak korzystać

💡 Czy wiesz, że...

Najczęściej zadawane pytania

Co znalazłeś?