Regression and Correlation

Regression fits a function of predictors to a response, while correlation measures the strength of a linear relationship. Least squares is the workhorse that connects them and links back to the projection ideas in Singular Value Decomposition.

Correlation

Covariance measures joint variation and correlation normalizes it (see Expectation and Moments):

$\rho =\frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}\in [-1,1]$

• Sample version: $r=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{{\sum }_i(x_i-\bar{x}{)}^2}\sqrt{{\sum }_i(y_i-\bar{y}{)}^2}}$.
• $r=\pm 1$ means points lie exactly on a line; $r=0$ means no linear trend.
• $r$ is invariant to shifts and positive scalings of $x$ or $y$.

Correlation is not causation

A high $r$ can arise from a confounder, reverse causation, or coincidence. Regression fits a line but does not establish that $X$ causes $Y$. Anscombe’s quartet shows four datasets with identical $r$ but wildly different shapes, so always plot the data.

Simple Linear Regression

Model the response as a line plus noise:

$y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i,{\epsilon }_i\sim \mathcal{N}(0,{\sigma }^2)\text{ i.i.d.}$

Derivation: Least-Squares Estimates

Least squares minimizes the residual sum of squares $S({\beta }_0,{\beta }_1)={\sum }_i(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2$. Setting the partial derivatives to zero gives the normal equations:

$\frac{\partial S}{\partial {\beta }_0}=-2{\sum }_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0,\frac{\partial S}{\partial {\beta }_1}=-2{\sum }_i{x}_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0.$

Solving yields

${\hat{\beta }}_1=\frac{{\sum }_i(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_i(x_i-\bar{x}{)}^2}=r\frac{s_y}{s_x},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}.$

The first normal equation forces the fitted line through the centroid $(\bar{x},\bar{y})$ and makes the residuals sum to zero.

Multiple Linear Regression

With $p$ predictors, stack the data into a design matrix $X\in {\mathbb{R}}^{n\times (p+1)}$ (a column of ones plus predictors):

$y=X\beta +\epsilon ,\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y.$

Derivation: minimizing $\Vert y-X\beta {\Vert }^2$ gives ${\nabla }_{\beta }=-2X^{\top }(y-X\beta )=0$, so $X^{\top }X\hat{\beta }=X^{\top }y$.

• This is the normal-equations solution, an Orthogonality and Projections of $y$ onto the column space of $X$: the fitted $\hat{y}=X\hat{\beta }$ is the closest point in that subspace and the residual is orthogonal to it.
• When $X^{\top }X$ is ill-conditioned, use the Singular Value Decomposition (giving the pseudoinverse) or ridge regularization $\hat{\beta }=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }y$.

Interpretation and Diagnostics

• ${\hat{\beta }}_j$: expected change in $y$ per unit change in predictor $j$, holding others fixed.
• Coefficient of determination: $R^2=1-\frac{{\text{SS}}_{\text{res}}}{{\text{SS}}_{\text{tot}}}$, the fraction of variance explained; in simple regression $R^2=r^2$.
• Test each coefficient with a $t$-test on ${\hat{\beta }}_j/\text{SE}({\hat{\beta }}_j)$ (see Hypothesis Testing).
• Check residual plots for the four assumptions: linearity, independence, constant variance (homoscedasticity), and normality.

Overfitting and extrapolation

$R^2$ never decreases when predictors are added, so use adjusted $R^2$ or cross-validation to compare models. Predictions far outside the observed range of $x$ are unreliable, and multicollinearity (correlated predictors) inflates coefficient variances and destabilizes signs.