Encoder-Decoder | AI Knowledge Hub

The Core Architecture for Sequence Transformations

The Encoder-Decoder architecture consists of two key components: an encoder that processes the input sequence and creates a representation, and a decoder that generates the output sequence based on these representations.

This architecture forms the foundation for tasks like machine translation, summarization, and other sequence transformation problems.

A Key Feature: Masked Attention for Autoregressive Generation

This architecture is frequently used in models such as GPT (Generative Pre-trained Transformer) and decoder-only models.

Masked attention plays a crucial role in connecting the encoder and decoder, enabling autoregressive generation.

live demo · related simulation● LIVE

The Outcome: High-Quality Translation

The process involves the encoder processing the input text and the decoder generating a summary based on this processed information.

Ultimately, this results in concise versions of texts that retain their core meaning.

Frequently asked questions

What is the Encoder-Decoder architecture?

The Encoder-Decoder architecture consists of an encoder and a decoder, where the encoder processes input sequences to create representations, and the decoder generates output sequences based on these representations.

How does cross-attention work?

Cross-attention allows the decoder to ‘look’ at the encoder outputs. The query (Q) is taken from the decoder, while the keys (K) and values (V) are taken from the encoder.

What role does cross-attention play in generating text?

Cross-attention enables the decoder to selectively attend to relevant parts of the encoded input sequence, allowing it to generate more accurate and contextually appropriate output.

▶ Try it live

Everything above runs in your browser — open Hash Function Avalanche Visualizer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Hash Function Avalanche Visualizer simulation