GroupNorm

Splits channels into $G$ groups and normalizes over $(C/G,H,W)$ within each sample, giving BatchNorm-like benefits without any batch dependence, the default for small-batch detection/segmentation.

Equation

For input $x\in {\mathbb{R}}^{N\times C\times H\times W}$ with group index $g$, let $S_g$ be the set of $(C/G)\cdot H\cdot W$ elements in group $g$ of sample $n$:

$${\mu }_{ng}=\frac{1}{|S_g|}\sum _{i\in S_g}{x}_i,{\sigma }_{ng}^2=\frac{1}{|S_g|}\sum _{i\in S_g}(x_i-{\mu }_{ng}{)}^2$$ $$y_i={\gamma }_c\cdot \frac{x_i-{\mu }_{ng}}{\sqrt{{\sigma }_{ng}^2+ϵ}}+{\beta }_c$$

where $\gamma ,\beta \in {\mathbb{R}}^C$ are per-channel affine parameters.

From scratch

import torch
from einops import rearrange
 
def group_norm(x, gamma, beta, groups, eps=1e-5):
    # x: (n, c, h, w); gamma, beta: (c,)
    xg = rearrange(x, "n (g k) h w -> n g (k h w)", g=groups)
    mu = xg.mean(-1, keepdim=True)
    var = xg.var(-1, keepdim=True, unbiased=False)
    xg = (xg - mu) * torch.rsqrt(var + eps)
    y = rearrange(xg, "n g (k h w) -> n (g k) h w", k=x.shape[1] // groups, h=x.shape[2])
    return y * gamma.view(1, -1, 1, 1) + beta.view(1, -1, 1, 1)

Notes

• $G=1$ recovers LayerNorm over $(C,H,W)$; $G=C$ recovers InstanceNorm.
• Identical behavior in train and eval: no running statistics.
• Requires $C\mathrm{mod}G=0$; $G=32$ is the usual default.
• See Normalization Comparison.