Decoding Focus: An Introduction to Attention Mechanisms

The Bottleneck Problem

Traditional recurrent neural networks (RNNs) like LSTMs and GRUs struggled with long sequences due to the ‘bottleneck’ problem. The entire input sequence was compressed into a fixed-size context vector, inevitably losing information about earlier parts of the sequence.

Introducing Attention: Weighted Context

Attention mechanisms address this by allowing the model to directly access all previous hidden states when processing a new input. Instead of a single context vector, attention assigns weights to each part of the input sequence based on its relevance.

α(i) = softmax(q(i); h_1, ..., h_T)

Self-Attention: Intra-Sequence Relationships

Self-attention, popularized by the Transformer architecture, calculates attention scores within a single sequence. Each element attends to all other elements in the same sequence, learning internal dependencies and relationships.

Attention(Q, K, V) = softmax((QK^T)/sqrt(d_k))V

Cross-Attention: Bridging Sequences

Cross-attention extends self-attention to multiple sequences. For example, in machine translation, one sequence (the encoder) attends to another (the decoder), allowing the decoder to focus on relevant parts of the source sentence.

Frequently asked questions

Try it live

Everything above runs in your browser — open SPH Fluid and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.