Mountain Car: SARSA(λ) with Eligibility Traces

Learning Parameters

Learning rate α 0.30

Exploration ε 0.10

Trace decay λ 0.90

Training speed 20×

Live Stats

Episode

Steps this episode

Avg steps (last 20)

—

Goals reached

How it works

The car sits in a valley between two hills and cannot climb the right hill on throttle alone — the agent must learn to reverse up the left slope first to build momentum. State is continuous: position x ∈ [−1.2, 0.6] and velocity v ∈ [−0.07, 0.07]. Actions are reverse / coast / forward.

v' = clip(v + 0.001·a + cos(3x)·(−0.0025))
x' = clip(x + v')
reward = −1 per step, 0 on reaching the goal (x ≥ 0.5)

Since the state space is continuous, Q-values live on a 20×14 tile grid over (position, velocity) — a simplified single-tiling stand-in for Sutton & Barto's tile coding. The agent is trained with SARSA(λ), which uses eligibility traces to spread each reward's credit back across the recently visited states, not just the last one:

δ = r + γ·Q(s',a') − Q(s,a)
e(s,a) ← 1                (replacing trace on the visited tile)
∀(s,a): Q(s,a) += α·δ·e(s,a)
∀(s,a): e(s,a) ← γλ·e(s,a)   (decay every step)

α learning rate — how much each TD error updates Q. Too high and learning is noisy; too low and it crawls.
ε exploration — probability of a random action instead of the greedy one, so the agent keeps discovering better routes.
λ trace decay — how far back credit for a reward propagates. λ=0 is one-step SARSA; λ near 1 credits a whole trajectory.

The purple bar field behind the car is the learned cost-to-go surface, −max_a Q(x,v,a), over every tile — tall where the agent still expects a long climb, flat once a state is well understood. Watch the "valley ridge" of tall bars erode as training progresses.

Car & hill Cost-to-go bars Current state

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.