Normalization Comparison

Side-by-side of the standard normalizers for a conv-shaped tensor $x\in {\mathbb{R}}^{N\times C\times H\times W}$ (transformers use $(N,T,D)$ with $D$ playing the role of $C$).

Equation

All share the same form, differing only in the reduction set $S$:

$$y_i=\gamma \cdot \frac{x_i-{\mu }_S}{\sqrt{{\sigma }_S^2+ϵ}}+\beta ,{\mu }_S=\frac{1}{|S|}\sum _{j\in S}{x}_j$$

Norm	Reduced axes	Stats shape	Batch-dep.	Train vs eval	Params
BatchNorm	$N,H,W$	$(C,)$	yes	differs: running $\mu ,\sigma$ at eval	$2C$
LayerNorm	$C,H,W$	$(N,)$	no	identical	$2C$
InstanceNorm	$H,W$	$(N,C)$	no	identical	$2C$
GroupNorm	$C/G,H,W$	$(N,G)$	no	identical	$2C$
RMSNorm	$C$ (no mean)	$(N,)$	no	identical	$C$

From scratch

import torch
 
def norm(x, dims, gamma, beta=None, center=True, eps=1e-5):
    # x: (n, c, h, w); dims: tuple of axes to reduce
    mu = x.mean(dims, keepdim=True) if center else 0.0
    var = ((x - mu) ** 2).mean(dims, keepdim=True)
    y = (x - mu) * torch.rsqrt(var + eps) * gamma.view(1, -1, 1, 1)
    return y + beta.view(1, -1, 1, 1) if beta is not None else y
 
# batch: dims=(0,2,3) | layer: (1,2,3) | instance: (2,3) | group: reshape to (n,g,-1) first

Notes

• Only BatchNorm couples samples, so it breaks at batch size 1, under heavy augmentation skew, and in distributed setups without SyncBN.
• $G=1\Rightarrow$ LayerNorm, $G=C\Rightarrow$ InstanceNorm: GroupNorm interpolates between them.
• RMSNorm drops centering, so it is scale- but not shift-invariant; Adaptive LayerNorm makes $\gamma ,\beta$ functions of a condition.
• All are $O(\text{numel})$ and memory-bandwidth bound; fusing the norm with the following matmul is the usual win.