Bayes Theorem and Conditional Probability

Conditional probability updates beliefs given evidence, and Bayes’ theorem reverses the direction of conditioning, turning a likelihood into a posterior. This is the engine of Bayesian inference and the counterpart to frequentist Maximum Likelihood Estimation.

Conditional Probability

For events with $P(B)>0$:

$P(A\mid B)=\frac{P(A\cap B)}{P(B)}$

• Multiplication rule: $P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)$.
• Chain rule: $P(A_1\cap \cdots \cap A_n)=P(A_1)P(A_2\mid A_1)\cdots P(A_n\mid A_1,\dots ,A_{n-1})$.

Conditioning on $B$ restricts the sample space to $B$ and renormalizes probabilities by $P(B)$.

Law of Total Probability

If $\{B_1,\dots ,B_n\}$ partition $\Omega$ (disjoint and exhaustive):

$P(A)={\sum }_{i=1}^{n}P(A\mid B_i)P(B_i)$

This decomposes an event over mutually exclusive scenarios and supplies the denominator in Bayes’ theorem.

Bayes’ Theorem

Starting from $P(A\cap B_i)=P(A\mid B_i)P(B_i)=P(B_i\mid A)P(A)$ and dividing by $P(A)$:

$P(B_i\mid A)=\frac{P(A\mid B_i)P(B_i)}{{\sum }_{j}P(A\mid B_j)P(B_j)}$

Term	Name	Role
$P(B_i)$	prior	belief before data
$P(A\mid B_i)$	likelihood	how well $B_i$ explains $A$
$P(B_i\mid A)$	posterior	belief after data
$P(A)$	evidence	normalizer

Base-rate trap (medical test)

A test is 99% accurate for a disease affecting 1 in 1000. Let $D$ be disease and $+$ a positive result: $P(+\mid D)=0.99$, $P(+\mid D^c)=0.01$, $P(D)=0.001$. Then $P(D\mid +)=\frac{0.99\cdot 0.001}{0.99\cdot 0.001+0.01\cdot 0.999}\approx \mathrm{0.09.}$ The low prior dominates, so about 91% of positives are false. Sensitivity alone is misleading without the base rate.

Independence

• Events: $A,B$ independent iff $P(A\cap B)=P(A)P(B)$, equivalently $P(A\mid B)=P(A)$.
• Conditional independence: $A,B$ independent given $C$ iff $P(A\cap B\mid C)=P(A\mid C)P(B\mid C)$.
• Pairwise independence does not imply mutual independence, and conditional independence neither implies nor is implied by marginal independence.

Intro to Bayesian Inference

Treat an unknown parameter $\theta$ as a random variable with prior $\pi (\theta )$. After observing data $x$:

$\pi (\theta \mid x)=\frac{f(x\mid \theta )\pi (\theta )}{\int f(x\mid {\theta }^{\prime })\pi ({\theta }^{\prime })d{\theta }^{\prime }}\propto f(x\mid \theta )\pi (\theta )$

• The posterior combines prior belief with the likelihood $f(x\mid \theta )$.
• Point estimates: posterior mean $E[\theta \mid x]$ or maximum a posteriori (MAP) $\arg {\max }_{\theta }\pi (\theta \mid x)$.
• A conjugate prior keeps the posterior in the same family (e.g. Beta prior with Binomial likelihood gives a Beta posterior).
• MAP with a flat prior coincides with the Maximum Likelihood Estimation estimate.

Prior sensitivity and the prosecutor's fallacy

Conclusions can hinge on the prior when data are scarce, so report sensitivity to it. Also do not confuse $P(A\mid B_i)$ with $P(B_i\mid A)$: a small $P(\text{evidence}\mid \text{innocent})$ does not mean a small $P(\text{innocent}\mid \text{evidence})$.