LSTM Memory Cell Gate Lab | mysimulator.uk

Why plain RNNs forget the past

A vanilla recurrent neural network updates its hidden state at every timestep by repeatedly multiplying by the same weight matrix and squashing the result through a nonlinearity like tanh. During training, the error signal used to adjust those weights must flow backward through every one of those multiplications, one per timestep. When the repeated factor is less than one, the gradient shrinks exponentially the further back it travels, a phenomenon called the vanishing gradient problem. In practice this means an RNN trained on a sentence or time series longer than a few dozen steps effectively cannot learn dependencies between an event near the start and an outcome near the end, because the training signal connecting them has decayed to nearly zero long before it reaches the relevant weights. The opposite failure, exploding gradients, can also occur when that factor is greater than one, causing wild, unstable updates.

The cell state: a conveyor belt for memory

LSTMs solve this by adding a second pathway called the cell state, which runs through every timestep with only minor, mostly linear modifications rather than repeated matrix multiplications and squashing. Picture it as a conveyor belt carrying information straight through the sequence; gradients can flow back along this belt almost unchanged, which is what keeps them from vanishing. The forget gate is the first thing information encounters at each step: a small neural layer looks at the current input and the previous hidden state, and outputs a value between zero and one for each piece of stored information, deciding how much of the old cell state to keep versus erase. A value near zero wipes that memory slot clean; a value near one preserves it untouched, letting critical information ride the belt indefinitely.

Writing and reading with the input and output gates

After forgetting happens, the input gate decides what new information gets written onto the belt. It combines a candidate update, computed from the current input, with a gate value that controls how much of that candidate actually gets added to the cell state, so irrelevant tokens can be mostly ignored while important ones are strongly incorporated. Finally the output gate controls what part of the updated cell state becomes the hidden state exposed to the rest of the network and to the next timestep. It filters the cell state through another learned gate value, so the LSTM can keep information stored internally for future use without necessarily broadcasting it as output right away. Together these three gates give the network fine-grained, learned control over memory: what to erase, what to add, and what to reveal.

Where LSTMs are used today

For roughly two decades LSTMs were the backbone of sequence modeling: language models, machine translation, speech recognition, handwriting generation, and time-series forecasting for anything from stock prices to sensor telemetry all relied on their ability to carry context across long gaps. Since 2017, transformer architectures with self-attention have overtaken LSTMs for most large-scale language and vision-sequence tasks because attention lets every position connect directly to every other position, parallelizes far better on modern hardware, and scales more predictably with data and compute. LSTMs remain popular, however, in resource-constrained or streaming settings, such as on-device keyword spotting, real-time control systems, and smaller time-series or sensor-fusion pipelines, where their lower memory footprint, strictly sequential processing, and strong inductive bias for ordered data are still genuinely useful.

Frequently asked questions

What exactly is the vanishing gradient problem, and why does it hurt plain RNNs so much?

During backpropagation through time, the error gradient used to update an RNN's weights is computed by repeatedly applying the chain rule across every timestep, which involves multiplying by the recurrent weight matrix and the derivative of the activation function again and again. When these repeated factors are smaller than one, which is common with saturating activations like tanh or sigmoid, the gradient shrinks exponentially as it travels backward through more timesteps. After enough steps it becomes numerically indistinguishable from zero, so the network receives essentially no learning signal connecting distant past inputs to the current error. Practically, this means a plain RNN can learn short-range patterns fine but fails to learn that, say, a subject at the start of a long sentence should agree with a verb dozens of words later, because that training signal never survives the backward pass.

How does the LSTM's cell state avoid the same vanishing gradient issue?

The cell state is updated mostly through elementwise addition and multiplication by gate values rather than repeated multiplication by a shared weight matrix followed by a saturating nonlinearity. When the forget gate outputs a value close to one, the cell state at the next step is nearly a direct copy of the previous one plus a small additive update, so the gradient flowing backward along this path is close to one as well instead of shrinking multiplicatively at every step. This near-linear, additive pathway is often described as a constant error carousel, and it lets gradients travel dozens or even hundreds of timesteps back with far less decay than in a vanilla RNN, which is exactly what allows LSTMs to learn long-range dependencies.

What do the forget, input, and output gates actually compute, and are they learned?

Yes, all three gates are small learned layers, typically a sigmoid function applied to a weighted combination of the current input and the previous hidden state, producing values between zero and one that act as soft on/off switches. The forget gate multiplies the previous cell state, controlling how much old information survives. The input gate multiplies a candidate new value before it is added to the cell state, controlling how much new information gets written in. The output gate multiplies a transformed version of the updated cell state to produce the hidden state that gets exposed as output. All the weights involved are trained jointly with the rest of the network via backpropagation, so the model learns for itself which information is worth keeping, adding, or revealing for the task at hand.

Try it live

Everything above runs in your browser — open LSTM Memory Cell Gate Lab and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open LSTM Memory Cell Gate Lab simulation