Sequence-to-Sequence Translation with Attention |…

Two sequences, one model

Machine translation, summarization, and speech transcription share an awkward property: the input and output are both variable-length sequences, and they rarely line up one-to-one. A five-word English sentence might become a seven-word French one, with the words in a different order entirely. Classic neural networks expect fixed-size inputs and outputs, so they cannot handle this directly. The sequence-to-sequence (seq2seq) architecture, introduced by Sutskever, Vinyals, and Le in 2014 and independently by Cho et al. the same year, solves this by splitting the job between two recurrent neural networks: an encoder that reads the entire source sequence and compresses it into a fixed-size vector, and a decoder that takes that vector and generates the target sequence one token at a time. Because the two halves are trained jointly end to end, the network learns its own internal representation of meaning without anyone hand-designing translation rules or alignment tables.

The encoder: reading and compressing

The encoder is typically an RNN, LSTM, or GRU that processes the source sentence one word at a time, updating its hidden state at every step so that it reflects everything seen so far. After the final word has been consumed, the encoder's last hidden state is treated as a summary of the whole sentence, often called the context vector. This single vector, usually just a few hundred numbers, is supposed to encode the sentence's meaning, structure, and every detail the decoder will need to reproduce it in another language. That is a lot to ask of one fixed-size vector, and it is precisely where the earliest seq2seq models started to struggle. As the source sentence gets longer, the encoder must cram more and more information into a container that never grows, so early words tend to get overwritten or diluted by the time the last word arrives, much like trying to summarize an entire novel in a single tweet regardless of how long the novel is.

The decoder: generating one token at a time

The decoder is itself an RNN, initialized with the encoder's final hidden state, that generates the target sentence one token at a time. At each step it takes its own previous hidden state and the previously generated word as input, produces a probability distribution over the target vocabulary, and picks the most likely next word, or samples from the distribution during more exploratory generation. That chosen word is then fed back in as input for the next step, so the decoder builds the output sequence autoregressively, word by word, until it produces a special end-of-sentence token. During training this process is guided by teacher forcing, where the true previous target word is fed in rather than the model's own guess, which stabilizes learning; at inference time the decoder must rely on its own predictions, which is why early mistakes can sometimes cascade into a garbled sentence.

Attention: stop squeezing, start looking back

The fixed-size context vector was the seq2seq architecture's Achilles' heel, and Bahdanau, Cho, and Bengio's 2014 paper fixed it with an idea now called attention. Instead of forcing the decoder to rely solely on one compressed summary, attention keeps every hidden state the encoder produced, one per source word, and lets the decoder consult all of them at every decoding step. At each step, the decoder computes an alignment score between its current state and each encoder hidden state, measuring how relevant that source word is right now. These scores are passed through a softmax to become attention weights that sum to one, and the encoder states are combined into a weighted sum, called the context, using those weights. Effectively the decoder gets to reach back and ask 'which source words matter most for the word I am about to produce,' rather than trusting a single vector to have remembered everything correctly. This is exactly why attention weights, when plotted as a heatmap, often trace a roughly diagonal line for languages with similar word order, or a distinctly bent one for languages that reorder phrases, such as adjectives moving before or after the noun.

From attention to the Transformer

Attention was originally bolted onto an RNN as a helpful add-on, but researchers soon noticed something more radical: the recurrence itself, the sequential loop that reads one word after another, was not what made attention useful. What mattered was the ability to relate any position in a sequence to any other position directly. In 2017, Vaswani et al. took that observation to its logical conclusion in the paper Attention Is All You Need, discarding recurrence entirely and building a model, the Transformer, out of nothing but attention layers and feed-forward networks. Removing the RNN's step-by-step dependency meant every position in a sequence could be processed in parallel during training rather than one token at a time, which was a massive win for speed on modern GPUs and TPUs. Every major large language model since, from BERT to GPT to today's frontier models, descends directly from that insight, making the humble encoder-decoder attention mechanism from machine translation one of the most consequential ideas in the history of deep learning.

Frequently asked questions

Why can't a plain seq2seq model just use a bigger context vector?

Making the context vector larger delays the bottleneck but does not remove it, and it also makes the model heavier and slower to train. The deeper problem is structural: no matter the size, a single fixed vector must represent an entire sentence regardless of length, so a thirty-word sentence and a three-word sentence are squeezed into containers of the same size. Attention sidesteps the problem altogether by keeping one hidden state per source word instead of merging them into one, so the amount of available information naturally scales with the length of the input.

What exactly are attention weights, and how are they learned?

Attention weights are a set of numbers, one per source word, that sum to one and indicate how much each source word should contribute to the decoder's current output step. They come from a small learned scoring function, often just a couple of neural network layers, that compares the decoder's current hidden state to each encoder hidden state and produces a raw compatibility score, which is then normalized with a softmax. Because this scoring function is just another set of weights inside the network, it is trained automatically through the same backpropagation process as the rest of the model, with no separate supervision telling it which words should align with which.

Does attention let the decoder see the whole sentence at once, unlike a plain RNN decoder?

Yes, in the sense that attention gives the decoder direct access to every encoder hidden state at every decoding step, rather than only whatever survived compression into one final vector. This is a big departure from a plain seq2seq decoder, which only ever receives the encoder's last hidden state. It is worth noting, though, that in classic RNN-based attention the encoder itself still reads the source sentence sequentially, so its individual hidden states already carry some left-to-right bias; it is really the Transformer's self-attention that removes sequential reading entirely.

Is the encoder-decoder with attention design still used today, or has it been fully replaced?

The specific combination of RNNs plus attention is largely retired in state-of-the-art systems, replaced by Transformer-based encoder-decoder or decoder-only models that use self-attention instead of recurrence. But the conceptual pattern, an encoder that builds a representation of the input and a decoder that attends to it while generating output, is still exactly how modern translation systems, image-captioning models, and speech-to-text systems are structured. Understanding the RNN-with-attention version remains one of the clearest ways to learn why attention works, before the added complexity of multi-head self-attention and positional encodings in a full Transformer.

What does the attention heatmap in the simulation actually represent?

Each row of the heatmap corresponds to one word the decoder is currently generating, and each column corresponds to one word in the source sentence; the brightness of a cell shows how strongly the decoder attended to that source word while producing that target word. A bright, roughly diagonal pattern usually indicates the two languages share similar word order, while bright cells that jump around off the diagonal reveal the model reordering words, for example moving an adjective from after the noun to before it, which is exactly the kind of behavior a fixed single context vector struggles to capture reliably.

Try it live

Everything above runs in your browser — open Sequence-to-Sequence Translation with Attention and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Sequence-to-Sequence Translation with Attention simulation