Hypothesis Testing

A hypothesis test decides between two competing claims about a parameter using data, quantifying how surprising the data would be if the default claim were true. It is the dual of the confidence interval built in Maximum Likelihood Estimation.

Null and Alternative

• Null hypothesis $H_0$: the default or “no effect” claim, e.g. $\mu ={\mu }_0$.
• Alternative $H_1$: what we suspect, e.g. $\mu \ne {\mu }_0$ (two-sided) or $\mu >{\mu }_0$ (one-sided).
• We either reject $H_0$ or fail to reject it; we never “accept” $H_0$, since absence of evidence is not evidence of absence.

Test Statistic and p-value

A test statistic measures the distance between the data and $H_0$. For a normal mean with known $\sigma$:

$Z=\frac{\bar{X}-{\mu }_0}{\sigma /\sqrt{n}},H_0:\mu ={\mu }_0.$

• The p-value is $P(\text{statistic at least as extreme as observed}\mid H_0)$.
• Reject $H_0$ when the p-value falls below the significance level $\alpha$ (commonly $0.05$).
• Equivalently, reject when $|Z|>z_{\alpha /2}$; the rejection region and the CI in Maximum Likelihood Estimation are duals.

Worked Example: One-Sample z-test

A process targets ${\mu }_0=50$ with known $\sigma =4$. A sample of $n=64$ gives $\bar{X}=51.2$. Then

$Z=\frac{51.2-50}{4/\sqrt{64}}=\frac{1.2}{0.5}=\mathrm{2.4.}$

The two-sided p-value is $2(1-\Phi (2.4))\approx 2\cdot 0.0082=0.0164$. Since $0.0164<0.05$ (and $2.4>z_{0.025}=1.96$), reject $H_0$: the mean has shifted.

What a p-value is not

The p-value is not $P(H_0\mid \text{data})$ and not the probability the result is due to chance. It is the tail probability of the data assuming $H_0$ is true. A large p-value does not prove $H_0$, and a small p-value does not measure effect size or practical importance.

Type I and Type II Errors

	$H_0$ true	$H_0$ false
Reject $H_0$	Type I error ($\alpha$)	correct (power)
Fail to reject	correct	Type II error ($\beta$)

• $\alpha =P(\text{reject }{H}_0\mid H_0)$ is the false-positive rate, chosen by the analyst.
• $\beta =P(\text{fail to reject}\mid H_1)$; power $=1-\beta$ rises with $n$, effect size, and $\alpha$.
• For fixed $n$, lowering $\alpha$ raises $\beta$: there is a tradeoff. Increasing $n$ lowers both.

Common Tests

Test	Use case	Statistic	Reference dist.
$z$-test	mean, known $\sigma$	$(\bar{X}-{\mu }_0)/(\sigma /\sqrt{n})$	standard normal
$t$-test	mean, unknown $\sigma$	$(\bar{X}-{\mu }_0)/(s/\sqrt{n})$	$t_{n-1}$
Two-sample $t$	compare two means	difference / pooled SE	$t$
Paired $t$	matched pairs	$t$ on differences	$t$
Chi-square	variance, goodness of fit, independence	$\sum (O-E{)}^2/E$	${\chi }^2$
ANOVA ($F$-test)	compare $\ge 3$ means	between/within variance	$F$

• The $t$-test replaces $\sigma$ with the sample standard deviation $s$ and uses $n-1$ degrees of freedom; its heavier tails account for estimating $\sigma$.
• Chi-square goodness-of-fit compares observed counts $O$ with expected counts $E$ across categories, with degrees of freedom equal to categories minus estimated parameters minus one.

Multiple comparisons and p-hacking

Running many tests inflates the family-wise error rate: at $\alpha =0.05$, about 1 in 20 true nulls is rejected by chance. Correct with Bonferroni ($\alpha /m$) or false-discovery-rate control, and pre-register hypotheses to avoid selecting tests after seeing the data.