Inside an LSTM Cell: How Forget, Input and Output Gates Remember the Right Things
A mechanics-level walkthrough of how LSTM gates decide what to keep, update and reveal at each time step, illustrated with an airline-passenger forecasting example.
Why plain recurrent networks forget too fast
A simple recurrent neural network passes a hidden state from one time step to the next, repeatedly multiplying it by the same weight matrix. Repeated multiplication is numerically unforgiving: if the dominant eigenvalue of that weight matrix is even slightly less than one, the hidden state shrinks toward zero over many steps and the gradient used to train earlier time steps vanishes along with it, so the network effectively can't learn dependencies that span more than a handful of steps. This is a serious limitation for problems like forecasting monthly airline passenger counts, where a meaningful pattern — a yearly seasonal cycle, say — spans twelve time steps or more. The Long Short-Term Memory (LSTM) architecture was designed specifically to give recurrent networks a mechanism for carrying information across long spans without that decay.
The cell state: a conveyor belt for information
The key structural addition in an LSTM is the cell state — a separate pathway that runs through the whole sequence largely undisturbed by direct nonlinear transformations. Instead of overwriting the cell state at every step the way a plain RNN overwrites its hidden state, the LSTM only adds to it or selectively erases parts of it. This is what makes it possible for the cell state to preserve information from ten, fifty, or more steps earlier without that information being crushed by repeated multiplication through squashing nonlinearities. Three learned gates — the forget gate, the input gate, and the output gate — control exactly what gets added, erased, and exposed at each time step.
The forget gate: deciding what to discard
At each time step, the forget gate looks at the current input and the previous hidden state, passes them through a sigmoid layer, and produces a vector of values between 0 and 1 — one value per unit in the cell state. A value near 0 means 'erase this piece of the memory,' and a value near 1 means 'keep it.' This vector is multiplied element-wise against the existing cell state, so the forget gate is the mechanism by which the network learns to drop information that's no longer relevant — for instance, once a seasonal peak has passed, the specific magnitude of that peak may no longer matter for predicting the next few months, and the forget gate can learn to fade it out while the network learns to hold onto the broader seasonal phase.
The input gate: deciding what new information to store
Two things happen in parallel to decide what new information enters the cell state. A sigmoid layer (the input gate proper) again outputs values between 0 and 1, this time deciding which parts of the cell state should be updated. Separately, a tanh layer produces a vector of candidate values — new information that could be added to the cell state, squashed into the range −1 to 1. The input gate's sigmoid output is multiplied against these candidate values, and the result is added to the cell state after the forget gate has done its erasing. This two-part design (deciding what could be added, and separately deciding how much of it to actually add) gives the network fine-grained control: it can propose a strong new memory but only partially admit it, or fully block it, depending on what the sigmoid gate has learned to detect as relevant.
The output gate: deciding what to reveal
The cell state itself is never output directly. Instead, the output gate — another sigmoid layer — decides which parts of the (now-updated) cell state should influence the hidden state and, by extension, the prediction made at this time step. The cell state is passed through a tanh function to squash it back into the −1 to 1 range, then multiplied by the output gate's sigmoid values to produce the new hidden state. This separation between an internal cell state (the long-term memory) and an externally visible hidden state (what gets used for prediction and passed to the next layer) is what lets an LSTM hold onto information it isn't ready to use yet, revealing it only at the time step where it becomes relevant.
Putting it together: a passenger-forecasting example
Consider a network built to forecast monthly airline passenger counts, taking one passenger count per time step as input and producing a linear-layer prediction from the final LSTM hidden state. Practical experiments with this kind of setup typically use a sliding-window approach, feeding the model a short history of prior values to predict the next one, and comparing hidden-layer sizes (say 50 units versus 100) to see whether extra capacity improves accuracy without overfitting the roughly one hundred available training points. A single-layer LSTM with a modest hidden size is often enough to noticeably outperform a naive 'predict the last value' baseline on this kind of smoothly trending, seasonal series, precisely because the gates let it retain a running sense of the trend and seasonal phase across steps rather than starting fresh at each prediction.
Practical levers when tuning an LSTM forecaster
A handful of design choices matter more than others when building an LSTM for a time-series task like this one. The lookback window — how many past observations feed into each prediction — trades off between giving the model more context and diluting the training signal when the dataset is small. Hidden size controls the network's representational capacity but risks overfitting on short series if pushed too high without regularisation. Stacking additional LSTM layers can capture more complex temporal patterns but multiplies the parameter count and the risk of vanishing signal through the extra layers. And because raw passenger counts can span a wide numeric range, scaling the input data (with something like min-max normalisation) before training is usually necessary for the gates' sigmoid and tanh activations to operate in their useful, non-saturated range.
Frequently Asked Questions
What's the difference between the cell state and the hidden state in an LSTM?
The cell state is the long-term memory pathway that gets selectively updated by the forget and input gates and can carry information across many time steps largely unmodified. The hidden state is the externally visible, filtered version of that memory — the output gate decides how much of the current cell state gets exposed as the hidden state used for predictions and passed on to the next layer or time step.
Why does an LSTM use both sigmoid and tanh activations?
Sigmoid outputs values between 0 and 1, which is exactly what's needed for a gate that decides 'how much of this should pass through' — 0 blocks it entirely, 1 lets it all through. Tanh outputs values between −1 and 1 and is used to produce or rescale the actual content of the memory, since a symmetric range around zero tends to keep the network's internal values numerically well-behaved during training.
Does a bigger hidden size always give a more accurate LSTM forecast?
Not necessarily. Increasing hidden size gives the network more capacity to model complex patterns, but on short time-series datasets it also raises the risk of overfitting to noise in the training data rather than learning the genuine seasonal or trend structure, so the improvement often shows up on training error while test error stalls or worsens.
Why is a sliding window used instead of feeding the whole time series at once?
A sliding window turns a single long sequence into many overlapping shorter training examples, giving the model far more training samples than the raw series alone would provide and letting it learn the relationship between a short recent history and the next value, which is the pattern that actually needs to generalise to future, unseen months.