🧠 Custom Hyperparameter Optimization Implementation

About this simulation

Every hyperparameter search — whether you write it yourself or use a library — is really an algorithm exploring a loss surface defined over two (or more) hyperparameter axes, trying to find the point of lowest validation loss without ever seeing the whole surface at once. This simulation renders that surface directly: height and colour both encode loss, and coloured search agents show how different custom optimizer implementations explore it.

How it works

• Random search — agents jump to a uniformly random point on every iteration, keeping the best result seen so far.
• Grid search — agents step through a fixed lattice of hyperparameter combinations in order, exhaustive but blind to the surface shape.
• Gradient descent (multi-start) — agents estimate the local slope by finite differences and step downhill, so several independent starts can each settle into different local minima.
• Surrogate-guided — agents bias new samples toward regions near the current best point (an exploration/exploitation trade-off), approximating how Bayesian optimization uses a cheap surrogate model to pick the next trial.

Controls

• Algorithm — switch the custom search strategy driving every agent.
• Trials / agents — how many parallel hyperparameter configurations are evaluated per iteration.
• Step size — how far each agent moves per iteration (learning rate / search radius).
• Landscape ruggedness — how many local minima the loss surface has, i.e. how hard the search problem is.
• 🎲 New landscape — regenerates the loss surface with a new random seed. ↺ Restart search — resets all agents and the best-loss tracker.

Did you know?

Real hyperparameter tuning libraries like Optuna and Hyperopt implement exactly this idea with a tree-structured Parzen estimator as the surrogate model — cheap to evaluate, used to decide which expensive real training run to try next.