Learned Optimizer Race: Meta-Learning to Descend a Loss Landscape

Task Landscape

Condition number κ 12.0

Learning rate α 0.35

Optimizers

SGD (hand-designed, fixed step) Momentum SGD (hand-designed) Learned Optimizer (meta-trained)

Live Loss (log₁₀)

SGD0.00

Momentum0.00

Learned0.00

Steps taken

First to converge (loss < 1e-3)

—

How it works

The task is an ill-conditioned quadratic bowl — the standard stress test used in learning-to-learn research:

f(x,y) = 0.5·(λ₁u² + λ₂v²)
u =  x·cosθ + y·sinθ
v = −x·sinθ + y·cosθ
κ = λ₂ / λ₁  (condition number)

A high κ means the bowl is a steep, narrow valley in one direction and almost flat in the other — exactly the geometry that makes fixed-step gradient descent zig-zag. Each optimizer updates its own position θ=(x,y) from the same gradient ∇f(θ):

SGD:       θ ← θ − α∇f(θ)

Momentum:  v ← 0.9v − α∇f(θ)
           θ ← θ + v

Learned:   s ← 0.95s + 0.05·∇f(θ)²
           θ ← θ − α·∇f(θ) / (√s + ε)

The "Learned Optimizer" rule is a per-coordinate adaptive step: it divides the gradient by a running estimate of its own recent magnitude, so steep directions get squashed and flat directions get boosted automatically. This is exactly the behaviour that "Learning to Learn by Gradient Descent by Gradient Descent" (Andrychowicz et al., 2016) found an LSTM meta-optimizer converges to after being meta-trained across many random quadratic tasks — the network's outer loop tunes a general update rule, and each new bowl is the inner loop the rule must solve without any further tuning. That is why the blue marker keeps heading toward the minimum on any κ, while red SGD zig-zags harder as κ grows.

κ slider — reshapes the bowl's anisotropy; higher κ is a harder task for hand-designed SGD.
α slider — the shared base learning rate every optimizer's rule is built on.
New Task — resamples the bowl's rotation θ and a fresh random start point, like drawing a new task from the meta-training distribution.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.