Loading 3D engineโ€ฆ

๐Ÿงฎ Hyperparameter Loss Landscape

Random Search
Online GD
Meta Warm-Start
Loss L(ฮป_t)
โ€“
Best L* found
โ€“
Regret_T
โ€“
Steps t
โ€“
How it works โ–พ
The bumpy surface is a synthetic hyperparameter loss L(ฮป) over two axes (log learning-rate, log regularization). A marker walks the surface using the selected strategy while every visited point is left behind as a trail. Regret_T = ฮฃt=1T [L(ฮป_t) โˆ’ L(ฮป*)] Random Search samples ฮป_t uniformly. Online GD follows โˆ’ฮทยทโˆ‡L(ฮป_t), perturbed by adversarial noise ฮด (robust optimization: minฮป maxฮด L(ฮป,ฮด)). Meta Warm-Start begins from a pre-learned good region โ€” ฮป*_new = MetaLearn({(D_i, ฮป*_i)}) โ€” then fine-tunes, so its regret grows far slower than the other two.
Drag to orbit ยท scroll to zoom