Layer Normalization: Per-Sample Feature Scaling (2D)

Activation Grid

Samples (rows) 4

Features per sample 8

Affine Transform (γ, β)

Scale γ 1.00

Shift β 0.00

Selected group

Row 1 of 4 (sample)

Raw mean μ

0.00

Raw variance σ²

0.00

Normalized mean

0.00

Normalized std

1.00

How it works

Layer Normalization (Ba, Kiros & Hinton, 2016) normalizes across the feature dimension of a single sample — never across the batch. That single difference from Batch Normalization is why it works for Transformers and RNNs, where batch statistics are unstable or unavailable at inference (e.g. generating one token at a time).

For one sample x = (x_1 … x_N):
μ  = (1/N) Σ x_i
σ² = (1/N) Σ (x_i − μ)²
x̂_i = (x_i − μ) / √(σ² + ε)
y_i = γ · x̂_i + β

Rows in the grid are independent samples (e.g. tokens); each row's cells are its own N features.
Axis: Per-Sample (LayerNorm) normalizes every row against itself — columns never interact.
Axis: Per-Feature (BatchNorm) flips the same formula onto each column instead — normalizing one feature across every sample in the batch, exactly what Batch Normalization does. Toggle it to see why LayerNorm is batch-size independent and BatchNorm is not: shrink the batch to 2 samples in BatchNorm mode and watch the normalized values swing wildly, while LayerNorm mode stays stable no matter how many rows there are.
Top band = raw activations. Bottom band = after normalization and the affine transform γ·x̂+β.
γ and β are the two learnable parameters every LayerNorm/BatchNorm layer keeps — they let the network undo the normalization if that minimizes the loss.

Real-world relevance: every Transformer block (GPT, BERT, ViT) wraps its attention and feed-forward sub-layers in LayerNorm to keep activations well-scaled through dozens of stacked layers, independent of sequence length or batch composition — unlike BatchNorm, which CNNs use instead because their batches are large and stable.

LayerNorm

Raw activation x_i Normalized y_i = γx̂_i+β