🧊 Why Big Starting Weights Can Stall a Neural Network

About this simulation

A sigmoid neuron computes z = w·x + b and squashes it through σ(z) = 1 / (1 + e⁻ᶻ). When weights start out large, even a modest input pushes z far from zero, landing the neuron in one of the sigmoid's flat saturated tails. In that region the curve barely moves — and the gradient used to update the weight during backpropagation, σ′(z) = σ(z)·(1 − σ(z)), collapses toward zero right along with it.

How it works

• The glowing curve is the sigmoid function; the orange ball marks the current pre-activation z and its output σ(z).
• The arrow at the ball is the local tangent — its length is proportional to σ′(z), the gradient available at that point. It shrinks to almost nothing out in the tails.
• The row of pillars simulates that same shrinking derivative multiplying through several stacked layers during backpropagation — the classic vanishing gradient problem. Pillar height fades toward the input side of the network.
• "Simulate gradient step" nudges the weight scale by a step proportional to the current gradient, so you can see how a saturated start barely moves at all — while a well-scaled start updates quickly.

Controls

• Weight scale — multiplies the weight in z = w·x; larger values push z further from zero.
• Input value (x) — the feature value feeding the neuron.
• Network depth — how many stacked sigmoid layers the gradient must flow back through.
• Show gradient arrow — toggles the tangent-line visualisation at the neuron.
• ▶ Simulate gradient step / ↺ Randomize input — see the "Did you know?" note below for why the step barely moves when saturated.

Did you know?

The sigmoid's derivative peaks at just 0.25, at z = 0. That means even in the very best case a sigmoid layer can pass along at most a quarter of the gradient it receives — stack a handful of such layers with poorly-scaled weights and the gradient reaching the earliest layers can shrink by many orders of magnitude, which is exactly why careful weight initialisation schemes (like Xavier/Glorot init) and alternative activations like ReLU became standard in deep learning.