AI PID Auto-Tuner: Reinforcement Learning for Industrial Process Control

Tank-Level Process

Setpoint (level) 60%

Time constant τ 6.0 s

Dead time L (sensor lag) 0.6 s

SPSA RL Auto-Tuner

Pair / Episode

0 / 0

Best ISE cost

—

Overshoot

—

Perturbation σ

25%

K_p = 2.00 K_i = 0.50 K_d = 0.10

How it works

The plant is a first-order-plus-dead-time (FOPDT) tank: a valve opens to position u(t) ∈ [0,120]%, and the level PV follows

τ · dPV/dt = K·u(t − L) − PV
u(t) = Kp·e(t) + Ki·∫e dt + Kd·de/dt,  e = SP − PV

τ is the tank's hydraulic capacitance/resistance, L is transport lag in the sensor line, and the integral term is anti-windup clamped while the valve is saturated. Each episode replays a full setpoint step and scores it with the classic auto-tuning objective:

Cost = ISE + 8·max(0, overshoot) + 0.02·∫u² dt
ISE  = ∫ e(t)² dt   (integral squared error)

The tuner is SPSA (Simultaneous Perturbation Stochastic Approximation) — a real reinforcement-learning-adjacent method used for industrial controller auto-tuning. Each pair of episodes perturbs the gain vector θ = (Kp,Ki,Kd) by one random ±1 direction Δ scaled by σ, evaluates the cost at θ+Δ and θ−Δ, and estimates a gradient from just those two rollouts:

ĝᵢ = (Cost(θ+Δ) − Cost(θ−Δ)) / (2·Δᵢ)
θ ← clamp(θ − a·ĝ),   σ and a decay every pair

The gain-space trail (right of the tank) drops one point per completed pair, colored green→red by cost — you can watch the agent walk toward the low-cost region of (Kp, Ki, Kd) space instead of a hand-tuned gain table.

idle

Level PV Setpoint Gain trail (Kp,Ki,Kd)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.