Maximum Likelihood Estimation

Point estimation guesses a parameter from data, and maximum likelihood picks the parameter that makes the observed data most probable. Interval estimation then reports the uncertainty around that guess.

Point Estimation

An estimator $\hat{\theta }=\hat{\theta }(X_1,\dots ,X_n)$ is a function of the sample used to guess a parameter $\theta$.

• Bias: $\mathrm{Bias}(\hat{\theta })=E[\hat{\theta }]-\theta$; unbiased means $E[\hat{\theta }]=\theta$.
• Consistency: ${\hat{\theta }}_n\to \theta$ in probability as $n\to \infty$.
• Mean squared error: $\mathrm{MSE}(\hat{\theta })=E[(\hat{\theta }-\theta {)}^2]=\mathrm{Var}(\hat{\theta })+\mathrm{Bias}(\hat{\theta }{)}^2$.

The sample mean $\bar{X}$ is unbiased for $\mu$; the sample variance $s^2=\frac{1}{n-1}{\sum }_i(X_i-\bar{X}{)}^2$ uses $n-1$ to stay unbiased for ${\sigma }^2$ (Bessel’s correction).

Method of Maximum Likelihood

Given i.i.d. data with density $f(x\mid \theta )$, the likelihood and log-likelihood are:

$L(\theta )={\prod }_{i=1}^{n}f(x_i\mid \theta ),\ell (\theta )={\sum }_{i=1}^n\log f(x_i\mid \theta )$

The MLE maximizes $\ell$ by solving the score equation and checking the second-order condition (see Optimality Conditions):

${\hat{\theta }}_{\text{MLE}}=\arg {\max }_{\theta }\ell (\theta ),\frac{\partial \ell }{\partial \theta }=0,\frac{{\partial }^2\ell }{\partial {\theta }^2}<0.$

Derivation: MLE for Common Distributions

Normal $\mathcal{N}(\mu ,{\sigma }^2)$. The log-likelihood is

$\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_i(x_i-\mu {)}^2.$

Setting $\partial \ell /\partial \mu =0$ gives $\hat{\mu }=\bar{X}$. Setting $\partial \ell /\partial {\sigma }^2=0$ gives ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(x_i-\bar{X}{)}^2$, which is biased low (divides by $n$, not $n-1$).

Bernoulli($p$). With $k={\sum }_i{x}_i$ successes, $\ell (p)=k\log p+(n-k)\log (1-p)$, so ${\ell }^{\prime }(p)=k/p-(n-k)/(1-p)=0$ gives $\hat{p}=k/n$, the sample proportion.

Exponential($\lambda$). $\ell (\lambda )=n\log \lambda -\lambda {\sum }_i{x}_i$, so ${\ell }^{\prime }(\lambda )=n/\lambda -{\sum }_i{x}_i=0$ gives $\hat{\lambda }=1/\bar{X}$.

Why maximize the likelihood

Under regularity conditions the MLE is consistent, asymptotically normal, and asymptotically efficient, attaining the Cramer-Rao lower bound $\mathrm{Var}(\hat{\theta })\ge 1/I(\theta )$ where $I(\theta )=-E[{\partial }^2\ell /\partial {\theta }^2]$ is the Fisher information. It is the frequentist counterpart to the MAP estimate in Bayes Theorem and Conditional Probability with a flat prior.

Interval Estimation

A confidence interval (CI) gives a range that covers $\theta$ with a stated frequency over repeated sampling.

• Normal mean, known $\sigma$: $\bar{X}\pm z_{\alpha /2}\frac{\sigma }{\sqrt{n}}$.
• Normal mean, unknown $\sigma$: $\bar{X}\pm t_{\alpha /2,n-1}\frac{s}{\sqrt{n}}$, using the $t$-distribution.
• Large-sample MLE CI: $\hat{\theta }\pm z_{\alpha /2}/\sqrt{I(\hat{\theta })}$.
• The half-width shrinks like $1/\sqrt{n}$, matching the Central Limit Theorem scaling.

Worked Example: Proportion CI

Out of $n=400$ users, $k=88$ convert, so $\hat{p}=0.22$. The standard error is $\sqrt{\hat{p}(1-\hat{p})/n}=\sqrt{0.22\cdot 0.78/400}\approx 0.0207$. A 95% CI is $0.22\pm 1.96\cdot 0.0207=(0.179,0.261)$.

Interpreting a 95% CI

A 95% CI does not mean ”$\theta$ is in this interval with probability 0.95.” $\theta$ is fixed and the interval is random. Over many samples, 95% of such intervals contain $\theta$. Also, MLEs can be biased in finite samples even though they are efficient asymptotically.