Central Limit Theorem

The laws of large numbers say sample averages converge to the mean, while the central limit theorem (CLT) says the fluctuations around that mean are normal regardless of the underlying distribution. Together they justify most of inferential statistics.

Laws of Large Numbers

For i.i.d. $X_1,\dots ,X_n$ with mean $\mu$, let ${\bar{X}}_n=\frac{1}{n}{\sum }_i{X}_i$.

• Weak LLN: ${\bar{X}}_n\to \mu$ in probability, so $P(|{\bar{X}}_n-\mu |>ϵ)\to 0$ for every $ϵ>0$.
• Strong LLN: ${\bar{X}}_n\to \mu$ almost surely.

The weak law follows from Chebyshev: $P(|{\bar{X}}_n-\mu |>ϵ)\le \mathrm{Var}({\bar{X}}_n)/{ϵ}^2={\sigma }^2/(n{ϵ}^2)\to 0$. The LLN pins down the limit but says nothing about the rate or the shape of the error.

CLT Statement

For i.i.d. $X_i$ with mean $\mu$ and finite variance ${\sigma }^2$:

$Z_n=\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}\stackrel{d}{\to }\mathcal{N}(0,1)\text{as }n\to \infty$

Equivalently, ${\sum }_i{X}_i\approx \mathcal{N}(n\mu ,n{\sigma }^2)$ for large $n$. Convergence in distribution means $F_{Z_n}(z)\to \Phi (z)$ at every $z$.

Intuition and Proof Sketch

• The scaling $\sqrt{n}$ is exactly what keeps fluctuations at a fixed size: the standard error $\sigma /\sqrt{n}$ shrinks, so we magnify by $\sqrt{n}$ to see a nondegenerate limit.
• Characteristic function argument (see Expectation and Moments): for the standardized variable $W_i=(X_i-\mu )/\sigma$, expand

${\phi }_W(t)=1-\frac{t^2}{2}+o(t^2),{\phi }_{Z_n}(t)={\left[{\phi }_W\left(\frac{t}{\sqrt{n}}\right)\right]}^n={\left(1-\frac{t^2}{2n}+o(1/n)\right)}^n\to e^{-t^2/2}.$

The limit $e^{-t^2/2}$ is the standard normal characteristic function, and convergence of characteristic functions implies convergence in distribution (Levy continuity).

Why the normal is everywhere

Any quantity that is a sum of many small independent effects is approximately normal. This is why measurement errors and aggregate noise tend to be Gaussian, and it does not depend on the shape of the individual terms.

Normal Approximation in Practice

• Binomial: $X\sim \text{Bin}(n,p)\approx \mathcal{N}(np,np(1-p))$ when $np$ and $n(1-p)$ are both $\gtrsim 10$. Use the continuity correction $P(X\le k)\approx \Phi ((k+0.5-np)/\sqrt{np(1-p)})$.
• Sample mean: build confidence intervals from ${\bar{X}}_n\pm z_{\alpha /2}\sigma /\sqrt{n}$ (see Maximum Likelihood Estimation).
• Rule of thumb: $n\ge 30$ is often enough unless the distribution is very skewed or heavy-tailed.

Worked Example: Dice Average

Roll a fair die $n=100$ times. Each roll has $\mu =3.5$ and ${\sigma }^2=35/12\approx 2.92$, so $\sigma \approx 1.71$. The sample mean is approximately $\mathcal{N}(3.5,2.92/100)$, giving standard error $0.171$. Thus $P({\bar{X}}_{100}>3.8)\approx 1-\Phi ((3.8-3.5)/0.171)=1-\Phi (1.76)\approx 0.039$.

The Delta Method

If $\sqrt{n}({\bar{X}}_n-\mu )\stackrel{d}{\to }\mathcal{N}(0,{\sigma }^2)$ and $g$ is differentiable at $\mu$ with $g^{\prime }(\mu )\ne 0$, then

$\sqrt{n}(g({\bar{X}}_n)-g(\mu ))\stackrel{d}{\to }\mathcal{N}(0,g^{\prime }(\mu {)}^2{\sigma }^2),$

extending asymptotic normality through smooth transformations.

When the CLT fails or is slow

Finite variance is required: heavy-tailed variables (e.g. Cauchy, infinite variance) do not obey the CLT. For strongly skewed data the approximation at $n=30$ can be poor, so check with a larger $n$ or a distribution-specific method.