Internal Covariate Shift

Internal covariate shift refers to the change in the distribution of a layer’s inputs during training, caused by the continual updates to all the parameters in the layers below it. It was the original motivation offered for BatchNorm.

Definition

Covariate shift classically means the input distribution changes between training and deployment. Internal covariate shift moves this idea inside the network: as lower-layer weights update each step, the distribution of activations feeding an upper layer keeps shifting. That layer must then continually re-adapt to a moving target, which the BatchNorm authors argued slows training and forces small learning rates.

$\text{ICS at layer }\ell :\text{the distribution of }{a}^{(\ell )}=f(W^{(\ell )}{a}^{(\ell -1)})\text{ changes as }{W}^{(<\ell )}\text{ updates}$

The BatchNorm Motivation

BatchNorm addresses this by standardizing each feature over the mini-batch, then applying a learnable scale $\gamma$ and shift $\beta$:

${\hat{x}}_i=\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}},y_i=\gamma {\hat{x}}_i+\beta$

By fixing the mean and variance of each layer’s inputs, the network below cannot arbitrarily shift the statistics seen above, which was claimed to reduce ICS and allow higher learning rates.

The Later Critique

Subsequent work (Santurkar et al., “How Does Batch Normalization Help Optimization?”) challenged the ICS story:

• They injected noise after BatchNorm to deliberately re-introduce distributional shift, yet training stayed fast. So reducing ICS was not what helped.
• They measured ICS directly and found BatchNorm did not clearly reduce it.

The loss-landscape view

The proposed explanation is that BatchNorm makes the optimization landscape smoother: it improves the Lipschitzness of the loss and its gradients (smaller, more predictable curvature). Smoother landscapes let larger, more stable steps be taken, which is what actually speeds up training.

What actually helps

Regardless of the mechanism debate, normalization (batch, layer, group) empirically stabilizes and accelerates training, reduces sensitivity to initialization and learning rate, and adds mild regularization. The takeaway is that the benefit is real; the “removing ICS” narrative is the part that did not hold up.

Illustration

import torch
 
x = torch.randn(64, 128) * 5 + 3   # activations with drifting scale/mean
 
mu = x.mean(dim=0, keepdim=True)
var = x.var(dim=0, unbiased=False, keepdim=True)
x_hat = (x - mu) / torch.sqrt(var + 1e-5)
 
print(x_hat.mean(0).abs().max().item())  # ~0
print(x_hat.var(0).mean().item())         # ~1
 
gamma = torch.ones(128)
beta = torch.zeros(128)
y = gamma * x_hat + beta   # learnable affine restores representational power

Normalizing over the batch axis (dim 0) forces every feature to zero mean and unit variance regardless of how the previous layer’s outputs drift. Contrast this with LayerNorm, which normalizes over the feature axis per example and is the norm of choice in Transformers where batch statistics are unstable.

Related

Internal covariate shift is the historical justification for BatchNorm; the practical alternative in sequence models is LayerNorm. Understanding why either helps ultimately comes down to how they reshape gradients during Backpropagation.