Policy Gradient CartPole — Evolution Strategies

Policy Training (Evolution Strategies)

Learning rate α 0.30

Exploration noise σ 0.40

Simulation speed 1×

Live Training State

Episode

Generation

This episode (s)

0.00

Best episode (s)

0.00

Avg reward (EMA)

0.00

‖w‖ (policy norm)

0.00

Current Policy

π(s)=tanh(w·s+b)

How it works

The controller is a single linear layer squashed by tanh — the simplest possible policy network — reading the cart-pole state s = (θ, θ̇, x, ẋ) and outputting a force:

F(s) = F_max · tanh(w·s + b)

Instead of backpropagating through the (non-differentiable, physics-simulated) episode, the weights are trained with Evolution Strategies (ES) — a zeroth-order policy-gradient method used in practice (e.g. OpenAI-ES) precisely because it needs no gradient of the environment, only episode rewards:

for each of N episodes this generation:
  sample noise  ε_i ~ N(0, I)
  test weights  w + σ·ε_i  → run 1 episode → reward R_i

advantage_i = R_i − mean(R)          (baseline, cuts variance)
Δw = (α / (N·σ)) · Σ_i advantage_i · ε_i
w  ← w + Δw

This is a Monte-Carlo estimator of the same objective policy-gradient methods like REINFORCE and PPO climb: ∇E[R] ≈ E[R·∇log π]. Subtracting the mean reward as a baseline is exactly the variance-reduction trick actor-critic methods use their critic for.

Cart-pole dynamics use the standard Barto–Sutton–Anderson equations for a pole hinged on a cart; an episode ends when |θ| exceeds ~40° or the cart leaves the track, and reward is the number of seconds balanced (capped at 20 s).

Learning rate α — step size of each weight update; too high causes the policy to overshoot and forget.
Noise σ — how far each generation explores around the current weights; too low stalls learning, too high makes updates noisy.
Reset — reinitialises w to small random values and starts training from scratch.

Cart + pole (live episode) Recent cart-position trail

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.