🎛️ Hyperparameter Constraints and Relationships

About this simulation

Hyperparameters are rarely independent. Changing one often forces another to change too — the classic example is the linear scaling rule: as mini-batch size grows, the learning rate should grow with it (roughly proportionally, or with a batch^α power law) to keep training stable. This scene renders a small search landscape over log(learning rate) and log(batch size). A curved corridor — the coupling constraint — is carved through the terrain in violet; configurations far from that corridor are treated as infeasible and shaded red.

How it works

• The terrain height is a synthetic loss bowl — lower is better, purely for depth cues.
• The violet ribbon is the constraint curve: ln(batch) ≈ base + α · ln(lr). Points near it satisfy the LR–batch-size relationship.
• Turning on the regularization-budget constraint adds a second, independent limit (dropout + weight-decay ≤ budget), and only configurations satisfying both constraints stay feasible — showing how joint constraints shrink the usable search space.
• Sampled points are random hyperparameter draws, colored by whether they fall inside the feasible intersection.

Controls

• Coupling strength (α) — the exponent in the LR–batch-size scaling relationship; reshapes the corridor's curvature and tilt.
• Constraint tolerance — how wide the feasible corridor is around the ideal relationship (a stricter vs. looser constraint).
• Sampled configs — how many random hyperparameter combinations to draw and test against the constraints.
• Add regularization-budget constraint — toggles a second, conditional constraint band that intersects the first.
• ↺ Resample configs — draws a fresh random batch of candidate configurations. Drag to rotate, scroll to zoom.

Did you know?

Tools like Optuna and Ray Tune let you encode exactly this kind of dependency — conditional search spaces where one hyperparameter's valid range depends on another's value — so the optimizer never wastes trials on combinations that were never going to work.