Expectation and Moments

Expectation is the probability-weighted average of a random variable, and moments, generating functions, and joint distributions summarize its shape and its relationships with other variables. These tools underpin estimation and the Central Limit Theorem.

Expectation

For a random variable $X$ (see Random Variables and Distributions):

$E[X]={\sum }_{x}x{p}_X(x)\text{(discrete)},E[X]={\int }_{-\infty }^{\infty }x{f}_X(x)dx\text{(continuous)}$

• Linearity: $E[aX+bY]=aE[X]+bE[Y]$ always, even when $X,Y$ are dependent.
• Law of the unconscious statistician: $E[g(X)]={\sum }_{x}g(x)p_X(x)$ or $\int g(x)f_X(x)dx$, with no need to find the distribution of $g(X)$.
• Monotonicity: $X\le Y$ pointwise implies $E[X]\le E[Y]$.

Variance and Higher Moments

• Variance: $\mathrm{Var}(X)=E[(X-\mu {)}^2]=E[X^2]-{\mu }^2$, with $\mu =E[X]$.
• Standard deviation: $\sigma =\sqrt{\mathrm{Var}(X)}$, in the same units as $X$.
• Affine scaling: $\mathrm{Var}(aX+b)=a^2\mathrm{Var}(X)$.
• $n$-th moment: $E[X^n]$; $n$-th central moment: $E[(X-\mu {)}^n]$.
• Skewness $E[(X-\mu {)}^3]/{\sigma }^3$ measures asymmetry; kurtosis $E[(X-\mu {)}^4]/{\sigma }^4$ measures tail weight.

Derivation: Variance Shortcut

Expand the definition using linearity:

$\mathrm{Var}(X)=E[(X-\mu {)}^2]=E[X^2-2\mu X+{\mu }^2]=E[X^2]-2\mu E[X]+{\mu }^2=E[X^2]-{\mu }^2.$

This is the identity used to read variances off the MGF below.

Moment Generating Functions

The moment generating function (MGF) packages all moments:

$M_X(t)=E[e^{tX}],E[X^n]=M_X^{(n)}(0)$

The name comes from the Taylor expansion $e^{tX}={\sum }_n\frac{t^n}{n!}{X}^n$, so the coefficient of $t^n/n!$ in $M_X(t)$ is $E[X^n]$.

• If the MGF exists in a neighborhood of $0$, it determines the distribution uniquely.
• The characteristic function ${\phi }_X(t)=E[e^{itX}]$ always exists and plays the same role, used to prove the Central Limit Theorem.
• For independent $X,Y$: $M_{X+Y}(t)=M_X(t)M_Y(t)$, since expectation factors over independent products.

Moments of the exponential

For $X\sim \text{Exp}(\lambda )$, $M_X(t)=\lambda /(\lambda -t)$ for $t<\lambda$. Then $M_X^{\prime }(0)=1/\lambda =E[X]$ and $M_X^{\prime \prime }(0)=2/{\lambda }^2$, so $\mathrm{Var}(X)=2/{\lambda }^2-1/{\lambda }^2=1/{\lambda }^2$.

Joint, Marginal, Conditional

For a pair $(X,Y)$ with joint PMF/PDF $p(x,y)$ or $f(x,y)$:

• Marginal: $f_X(x)=\int f(x,y)dy$ (sum over $y$ if discrete).
• Conditional: $f_{Y\mid X}(y\mid x)=\frac{f(x,y)}{f_X(x)}$ for $f_X(x)>0$.
• $E[Y\mid X=x]=\int y{f}_{Y\mid X}(y\mid x)dy$.
• Tower rule: $E[E[Y\mid X]]=E[Y]$.
• Law of total variance: $\mathrm{Var}(Y)=E[\mathrm{Var}(Y\mid X)]+\mathrm{Var}(E[Y\mid X])$.

Covariance and Independence

• Covariance: $\mathrm{Cov}(X,Y)=E[(X-{\mu }_X)(Y-{\mu }_Y)]=E[XY]-E[X]E[Y]$.
• Correlation: $\rho =\frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}\in [-1,1]$ (see Regression and Correlation).
• Sum rule: $\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\mathrm{Cov}(X,Y)$.

$X,Y$ are independent iff $f(x,y)=f_X(x)f_Y(y)$. Independence implies $\mathrm{Cov}(X,Y)=0$ and $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$.

Zero covariance is not independence

Uncorrelated ($\rho =0$) means no linear relationship, but variables can be dependent nonlinearly (e.g. $Y=X^2$ with $X$ symmetric about $0$ has $\rho =0$). The converse (independent implies uncorrelated) always holds.