🎯 Setting Initial Hyperparameter Values

About this simulation

Every training run starts with a choice of initial weight values before a single gradient step is taken. This scene represents a simplified loss landscape as a 3D surface: low points (valleys) are good parameter settings, the deepest gold-marked valley is the global minimum, and the surrounding bumps are local minima that can trap an optimizer. Each glowing particle is one training run, placed on the surface according to the chosen initialization strategy, then descending the gradient step by step — just like gradient descent updating a model's weights.

How it works

• Zero init — every particle starts at the exact same spot. In real networks this causes symmetric neurons to learn identical, redundant features; here it shows every run taking an identical, often stalled path.
• Large constant — particles start far out near the landscape's edge, mirroring oversized initial weights that push activations into saturated, hard-to-train regions.
• Naive wide-uniform — particles scatter randomly across a wide area, so some get lucky and some fall into distant local minima.
• Xavier/Glorot and He init — variance-scaled strategies that place particles in a moderate band around the center, balancing exploration with a good chance of reaching the global minimum, similar to how these schemes keep signal variance stable across layers.

Controls

• Init strategy — how the starting "weights" (particle positions) are chosen.
• Learning rate — step size each particle takes along the negative gradient each tick.
• Landscape ruggedness — how deep and numerous the local minima are, i.e. how non-convex the loss surface is.
• Particle count — how many independent training runs to simulate at once.
• ↺ Re-initialize & run — redraws the landscape and restarts every particle from a fresh sample of the chosen strategy.

Did you know?

Before Xavier Glorot & Yoshua Bengio's 2010 initialization scheme and Kaiming He's 2015 follow-up for ReLU networks, deep nets were notoriously hard to train simply because naive random initialization made gradients vanish or explode within a few layers — no amount of tuning the learning rate could fix a bad starting point.