Home▸AI & Machine Learning▸Transformer Feed-Forward Block: Per-Token Expand & Project

Transformer Feed-Forward Block: Per-Token Expand & Project

Every transformer block hides a second sublayer besides attention: a position-wise feed-forward network that expands each token's vector, applies a nonlinearity, and projects it back down — independently, with no mixing across tokens. Watch it run in 3D with live parameter counts and sparsity readouts.

AI & Machine Learning3DAdvanced60 FPS📱 Mobile-adapted⇄ 2D version

Attention gets most of the attention, but roughly two-thirds of a transformer's parameters live in its other sublayer: the position-wise feed-forward network. This simulator renders every token's vector as a column of bars and runs it through the real two-layer MLP — expand to d_ff, apply GELU or ReLU, project back to d_model, add the residual — independently for each position, with live parameter counts and per-token sparsity readouts so you can see exactly why this sublayer, not attention, holds most of the weights in models like GPT and BERT.

⚙ Under the hood

Every transformer block hides a second sublayer besides attention: a position-wise feed-forward network that expands each token's vector, applies GELU or ReLU, and projects it back down — independently, with no mixing across tokens. Watch it run in 3D with live parameter counts and activation-sparsity readouts.

transformerfeed-forwardneural-networkdeep-learninggeluattention

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install