Home▸AI & Machine Learning▸Why Big Starting Weights Can Stall a Neural Network Before It Learns Anything

🧊 Why Big Starting Weights Can Stall a Neural Network Before It Learns Anything

A close look at sigmoid saturation and the vanishing-gradient problem it triggers, using a simple logistic-regression-from-scratch example to show why weight initialisation is not a minor detail.

AI & Machine Learning3DAdvanced60 FPS

A 3D sigmoid curve with a live neuron marker shows exactly why oversized starting weights push a network into its flat, saturated tails — and a chain of gradient pillars shows that same collapsing slope multiplying away through the layers of backpropagation.

🔬 What It Demonstrates

The ball's position on the curve is the neuron's pre-activation value and output; the tangent arrow's length is the local gradient. As weight scale grows, the ball slides into the flat tails and the arrow shrinks toward nothing — the vanishing-gradient problem made visible.

🎮 How to Use

Increase the weight-scale slider to watch the neuron saturate, adjust the input value, pick how many layers the gradient must travel through, and press "Simulate gradient step" to see how little a saturated weight actually moves on one update.

💡 Did You Know?

The sigmoid's derivative peaks at only 0.25, at z = 0 — meaning even under ideal conditions each sigmoid layer can pass along at most a quarter of the gradient it receives, which is why deep networks moved toward ReLU-family activations and careful weight initialisation schemes.

⚙ Under the hood

A close look at sigmoid saturation and the vanishing-gradient problem it triggers, using a simple logistic-regression-from-scratch example to show why weight initialisation is not a minor detail.

neural networksmachine learningvanishing gradientsweight initializationlogistic regressionoptimizationdeep learningThree.js

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install