Home▸AI & Machine Learning▸Transformer Feed-Forward Block (2D): Per-Token Expand & Project

Transformer Feed-Forward Block (2D): Per-Token Expand & Project

A 2D bar-chart and heatmap view of a transformer's position-wise feed-forward sublayer: watch a token's vector expand through W1, pass through GELU or ReLU, and project back through W2 with a residual add — plus a live activation-sparsity heatmap across every token and a plot of the nonlinearity itself.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Attention gets most of the attention, but roughly two-thirds of a transformer's parameters live in its other sublayer: the position-wise feed-forward network. This 2D simulator renders every token's vector as a column of bars and runs it through the real two-layer MLP — expand to d_ff, apply GELU or ReLU, project back to d_model, add the residual — independently for each position. A draggable hidden-unit heatmap and an activation-curve plot make the per-token sparsity and the shape of the nonlinearity visible at a glance.

⚙ Under the hood

transformerfeed-forwardneural-networkdeep-learninggeluattention

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install