Why Big Starting Weights Can Stall a Neural Network Before It Learns Anything
A close look at sigmoid saturation and the vanishing-gradient problem it triggers, using a simple logistic-regression-from-scratch example to show why weight initialisation is not a minor detail.
A model that refuses to improve
Imagine training the simplest possible neural network — a single logistic regression unit predicting whether an apple harvest will exceed 80 fruit, based on temperature, rainfall and humidity — and finding that the loss barely moves, epoch after epoch, even though the model is clearly capable of fitting this data. This isn't a bug in the training loop. It's a direct, reproducible consequence of how the sigmoid activation function behaves at extreme inputs, combined with how its weights happened to be initialised. Understanding why requires looking closely at the shape of the sigmoid curve and what happens to it during backpropagation.
The sigmoid function and its flat tails
The sigmoid function squashes any real-valued input into the range between 0 and 1, which is exactly what makes it useful for producing a probability-like output in binary classification. But that squashing has a cost: for large positive or large negative inputs, the sigmoid curve is nearly flat. A big change in the input produces almost no change in the output once the input has moved far enough from zero — the function has 'saturated.' Mathematically, the derivative of the sigmoid function is largest (0.25) at an input of zero and shrinks toward zero as the input moves toward either extreme. That derivative is exactly the quantity that gets multiplied through the chain rule during backpropagation to compute how much the loss should change the weights.
How large initial weights cause saturation
If a network's weights are initialised with large random values, the very first forward pass computes weighted sums that can land far out in the sigmoid's flat tails, even before any training has occurred. The output might already be very close to 0 or 1 — sometimes matching the correct label by chance, sometimes not — but either way, the local derivative of the sigmoid at that point is close to zero. Because that near-zero derivative gets multiplied into every gradient computed for the weights and bias, the gradients themselves come out near zero, and the weight-update step (proportional to the gradient) is correspondingly tiny. The network isn't wrong so much as stuck: its loss can be high, but the gradient signal telling it how to fix that loss has effectively vanished.
Fixing it: small, careful initialisation
The direct fix demonstrated in a from-scratch logistic regression implementation is to shrink the initial weights — for instance, drawing them from a small random distribution and scaling that draw down by a factor of a thousand before training starts. This keeps the initial weighted sums close to zero, which is precisely the region where the sigmoid's derivative is largest and gradients can flow freely. With small initial weights, the same architecture that previously stalled can converge to near-perfect training accuracy within a modest number of epochs, illustrating that the model's capacity to learn was never the issue — only the numerical starting point was.
Beyond this toy example: why deep networks need principled initialisation
This effect compounds in deeper networks. Each additional sigmoid or tanh layer multiplies another derivative term (each less than or equal to 0.25 for sigmoid) into the chain-rule product that determines the gradient reaching earlier layers, so even with reasonably-scaled weights, gradients can shrink exponentially with depth — the broader vanishing-gradient problem that made very deep sigmoid-based networks difficult to train for years. This is part of why modern deep networks favour ReLU-family activations, whose derivative is a constant 1 for positive inputs and doesn't saturate on that side, and why principled initialisation schemes such as Xavier/Glorot and He initialisation were developed: they choose the initial weight scale as a function of the number of inputs and outputs of each layer specifically to keep the variance of activations, and therefore the size of gradients, roughly stable as signals pass forward and backward through the network.
Leaf tensors and why in-place weight updates matter too
A related, easily-overlooked detail when implementing gradient descent by hand in a framework like PyTorch is how weight updates are applied. Updating a parameter tensor using its underlying `.data` attribute for an in-place operation keeps that tensor as a 'leaf' node in the computation graph, which is what allows the automatic differentiation engine to correctly accumulate gradients on it at the next step. Get this wrong — for instance, by replacing the weight with a fresh tensor produced through an operation that's tracked by autograd — and the optimisation can silently break in ways that look, at first glance, exactly like a vanishing-gradient problem: the loss stalls, but for a bookkeeping reason rather than a mathematical one. Distinguishing between the two requires checking whether gradients are actually being computed and are simply tiny (saturation) versus not being computed or applied at all (a graph-tracking bug).
Frequently Asked Questions
Why does the sigmoid function cause vanishing gradients specifically at its tails?
The sigmoid's derivative is largest, at 0.25, when its input is zero, and shrinks toward zero as the input grows large in either the positive or negative direction. Since backpropagation multiplies this derivative into the gradient computation via the chain rule, an input far out in either tail produces a near-zero derivative and therefore a near-zero gradient, regardless of how large the actual loss is.
Does scaling down initial weights fully solve the vanishing gradient problem?
It solves the specific failure mode of accidental saturation from oversized initial weights in a shallow model. In deeper networks, gradients can still shrink across many layers even with well-scaled starting weights, because each sigmoid or tanh layer's derivative caps out below 1 and those factors compound multiplicatively with depth. That broader problem is addressed with additional tools like ReLU-family activations, residual connections, and principled initialisation schemes such as Xavier or He initialisation.
Why does the model sometimes give correct predictions even while stuck with a saturated sigmoid?
A saturated output near 0 or 1 can coincidentally match the correct label just by chance from the random initial weights, so predictions may look fine even though the loss and gradients show the model isn't actually learning anything from the data. This is a useful diagnostic: a model whose accuracy looks reasonable but whose loss curve is flat across epochs is a strong sign of gradient saturation rather than of genuine convergence.
Is Xavier or He initialisation better than just using small random weights?
For simple, shallow models, uniformly small random weights are often sufficient to avoid saturation, as in a basic logistic regression example. For deeper networks, Xavier and He initialisation are generally preferred because they scale each layer's initial weight variance based on that layer's number of inputs and outputs, keeping signal variance roughly consistent across many layers rather than requiring one arbitrary global scale-down factor to happen to work.