🗺️ Search Space Analysis

About this simulation

Every model has hyperparameters — learning rate, regularization strength, tree depth and so on — and tuning them means searching a multi-dimensional hyperparameter space. This scene collapses two hyperparameters (x₁, x₂) onto a horizontal plane and plots the resulting validation loss as a 3D surface: peaks are bad configurations, valleys are good ones. The surface's topology — how many valleys it has, how smooth or jagged it is — determines how hard the space is to search.

How it works

• Grid search — evaluates a uniform lattice of points; exhaustive but its cost explodes as dimensions grow (the "curse of dimensionality").
• Random search — samples uniformly at random; often finds a good region faster than grid search in high dimensions.
• Latin hypercube — stratifies each axis into equal bins so samples spread evenly across the space with less clustering than pure random.
• Bayesian-guided — starts broad, then concentrates new samples around the best point found so far, shrinking its search radius as evidence accumulates.
• Spheres are colored from teal (low loss) to purple/magenta (high loss); the glowing marker tracks the best configuration found so far.

Controls

• Search strategy — choose how new hyperparameter configurations are proposed.
• Sample budget — total evaluations allowed before the search stops.
• Landscape ruggedness — how many local minima/maxima the loss surface has, i.e. how non-convex the space is.
• Sampling speed — how fast new points are evaluated and drawn.
• Show surface wireframe — overlay the mesh grid to see the surface's structure more clearly.

Did you know?

In high-dimensional hyperparameter spaces most of a hypercube's volume sits near its corners and surface, not its center — a geometric quirk that is part of why naive grid search scales so poorly and why smarter samplers like Bayesian optimization matter.