Setting Initial Hyperparameter Values: Gradient-Descent Landscape (2D)

Initial Point θ₀

Initial x -2.20

Initial y -0.60

Optimizer Hyperparameters

Learning rate (lr) 0.30

Momentum (β) 0.40

Live Readout

Iteration

Loss L(θ)

0.000

‖∇L‖

0.000

Status

descending

Reached

—

How it works

The loss surface is hand-defined as a small quadratic bowl (pulls every point gently toward the origin) minus three Gaussian wells of different depth — a shallow local minimum, a medium one, and a deeper global minimum:

L(x,y) = 2.2 − Σᵢ aᵢ·exp(−‖(x,y)−(xᵢ,yᵢ)‖²/2sᵢ²) + 0.045·(x²+y²)

Each animation frame runs the real momentum gradient-descent update on the analytic gradient of that formula:

v ← β·v − lr·∇L(θ)
θ ← θ + v

This is exactly why initial values matter: the same optimizer with the same hyperparameters lands in whichever basin θ₀ starts inside — a point started near the shallow well stays stuck there even though a deeper minimum exists elsewhere on the surface. Push the learning rate high enough and the update overshoots the curvature of a well and the trajectory diverges outward instead of settling; push it too low and hundreds of iterations barely move the point.

Heatmap — darker/cooler = lower loss (better); the three wells are visible as dark basins.
Trail — the descent path taken by θ so far, fading with age.
Loss chart — L(θ) per iteration; a flattening curve is convergence, a curve rocketing upward is divergence.

Descent trajectory Global minimum Local minima