3D Robot Arm: Q-Learning Reaching Simulator

Q-Learning Robot Arm

Learning rate α 0.30

Discount γ 0.90

Exploration decay 0.985

Steps per frame 3

Episode

–

Epsilon ε

–

Dist to target

–

Reach rate

–

How it works

The arm has two joints — shoulder angle θ1 and elbow angle θ2 — each discretized into 12 bins, giving 144 possible states. Four actions nudge one joint up or down by one bin. Every step the agent picks the greedy action from its Q-table (or a random one with probability ε), moves, and measures how much closer the end-effector got to the target sphere. That change becomes the reward, and one Q-table entry updates with the real rule:

Q(s,a) += α · ( reward + γ · max(Q(s′,a′)) − Q(s,a) )

Q-bars — the floating grid beside the arm is a live 3D bar chart of max Q-value per (θ1, θ2) state; taller, brighter bars mean the agent is confident that joint configuration is valuable.
α / γ / decay — same roles as in tabular Q-learning generally: update strength, how far ahead reward is valued, and how fast random exploration gives way to the learned policy.
New target — moves the goal sphere without touching the Q-table, so you can watch a partly-trained policy generalize (or fail to) to an unseen target.

Discretizing continuous joint angles into bins is exactly how classic tabular RL was first applied to real manipulators, before function approximation (deep Q-networks) made continuous state spaces tractable.

Arm Target Q-value bars

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.