🖥️ Parallel and Distributed Hyperparameter Optimization

About this simulation

The glowing grid is a 2D hyperparameter search space (think learning rate on one axis, batch size or regularization on the other) rendered as a landscape where height and colour encode validation loss — green valleys are good configurations, red peaks are bad ones. A fleet of worker machines orbiting above the surface each fire trial "probes" down onto the landscape, and a central scheduler node collects their results and coordinates what gets tried next, exactly as a real distributed tuning job (Ray Tune, Optuna, SageMaker HPO, Vertex AI Vizier…) spreads trials across a cluster.

How it works

• Random search — every worker samples points uniformly at random, with no communication needed between trials.
• Bayesian-guided — workers increasingly bias new samples toward the current best-known region reported by the scheduler, shrinking their search radius over time (exploitation vs. exploration).
• ASHA (early-stopping) — trials that look worse than the recent median are pruned early (small red flashes) instead of run to completion, freeing compute for promising configurations, as in Successive Halving / Hyperband.
• Synchronous — every worker must finish, report, and receive the updated global best before any worker starts its next round: a slow "straggler" worker stalls the whole fleet.
• Asynchronous — each worker reports and starts its next trial independently, so stragglers no longer block the others, at the cost of workers sometimes acting on slightly stale information.

Controls

• Workers — number of parallel machines searching the space (2–16).
• Search strategy — how each worker chooses its next hyperparameter configuration.
• Sync mode — whether workers wait for a shared barrier or run fully independently.
• Network latency — simulated round-trip delay for worker ↔ scheduler communication.
• ↺ Reset search — clears all trials and starts a fresh search. Drag to rotate, scroll to zoom.

Did you know?

Because compute nodes rarely finish at the same time, purely synchronous distributed search wastes a large fraction of cluster time waiting on the slowest worker — this is exactly why systems like ASHA and asynchronous Hyperband were designed to overlap scheduling decisions with still-running trials instead of waiting for a clean round boundary.