Every transformer block has two sublayers. Self-attention mixes information across token positions. The second sublayer — the position-wise feed-forward network (FFN) — does the opposite: it transforms each token's vector independently, applying the exact same two-layer MLP to every position with no cross-token interaction at all:
FFN(x) = W2 · φ(W1·x + b1) + b2
x ∈ ℝ^d_model (one token's vector)
W1 ∈ ℝ^(d_ff × d_model), b1 ∈ ℝ^d_ff — expand
φ — nonlinearity (GELU or ReLU)
W2 ∈ ℝ^(d_model × d_ff), b2 ∈ ℝ^d_model — project back
output = x + FFN(x) — residual connection
dff is usually 4× wider than dmodel (this simulator's default) — real models like GPT-2 use 768→3072→768, GPT-3 uses 12288→49152→12288 per token, per layer. That expand-then-compress pattern gives the network extra capacity to recombine features before shrinking back to the residual stream's width.
- Top panel — four bar columns for the active token: input x, the raw expanded pre-activation W1x+b1, the same vector after the nonlinearity φ, and the final output x+W2φ(·)+b2. Blue = positive, orange = negative.
- Middle panel — hidden-unit heatmap: every row is a token, every column a hidden unit (after the nonlinearity). Drag horizontally to pan when dff is wide, or click a row to jump straight to that token.
- Bottom panel — the nonlinearity itself: GELU vs ReLU plotted as a curve, with a dot for every one of the active token's pre-activation values marking exactly where it lands on the curve.
- GELU vs ReLU — ReLU hard-zeros every negative unit (a sharp cliff at 0); GELU only softly damps small negative inputs, which is why modern transformers (GPT, BERT, ViT) mostly use GELU or its variants.
- Because every position runs through the same W1, b1, W2, b2, this sublayer is often described as "a 1×1 convolution applied to every token" — it has zero notion of sequence order or neighboring tokens; that's entirely attention's job.