How it works โพ
The bumpy surface is a synthetic hyperparameter loss L(ฮป) over two axes
(log learning-rate, log regularization). A marker walks the surface using the
selected strategy while every visited point is left behind as a trail.
Regret_T = ฮฃt=1T [L(ฮป_t) โ L(ฮป*)]
Random Search samples ฮป_t uniformly. Online GD follows
โฮทยทโL(ฮป_t), perturbed by adversarial noise ฮด (robust optimization:
minฮป maxฮด L(ฮป,ฮด)). Meta Warm-Start begins from a
pre-learned good region โ ฮป*_new = MetaLearn({(D_i, ฮป*_i)}) โ
then fine-tunes, so its regret grows far slower than the other two.