Hindsight Experience Replay: Robot Reaching with Sparse Rewards

Twin Learning Agents

Hindsight Experience Replay

Baseline (sparse reward only)

Learning rate α 0.15

Exploration σ 0.25

Sim speed (steps/s) 10

Live Progress

Episodes run

Success gap

+0%

HER success (last 40)

Baseline success (last 40)

success rate over episodes0–100%

How it works

Both arms are 2-link manipulators using the exact same architecture — a linear-Gaussian policy π(a|s) = 𝒩(θ·s + b, σ²) where s is the direction from the end-effector to the goal. Both learn from the same sparse reward: r = 1 only if the arm reaches within threshold of the goal by the end of the episode, otherwise r = 0.

REINFORCE update (both arms):
θ ← θ + α (G − b̄) Σₜ ∇θ log π(aₜ|sₜ)
∇θ log π(a|s) = (a − μ)/σ² · sᵀ

G = 1 if goal reached, else 0
b̄ = running average return (baseline)

With a sparse reward, most random episodes fail completely — G = 0 for almost every rollout early on, so the baseline arm gets almost no gradient signal and learns very slowly.

Hindsight Experience Replay ("final" strategy,
Andrychowicz et al. 2017):

After a failed episode, relabel the goal with
the position the arm ACTUALLY reached:
  g' = end-effector position at final step

Because g' was reached by construction, this
relabeled episode always looks successful —
giving the HER arm a usable learning signal
on every single episode, not just the rare
real successes.

Learning rate — step size of the policy-gradient update.
Exploration σ — standard deviation of the Gaussian action noise; too low and neither arm ever stumbles onto a goal, too high and the update is dominated by noise.
Reset weights — restarts both policies from small random weights so you can watch the success-rate gap open up again from scratch.

This is a simplified, browser-sized version of the real mechanism behind HER-based robotic manipulation training (used in real sim-to-real reaching and grasping pipelines): identical optimizer, identical reward sparsity, the only difference is whether failed trajectories get reinterpreted as successes toward a different goal.

HER arm — reach target Baseline arm — reach target

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.