Bias-Variance Tradeoff

The bias-variance tradeoff explains why expected prediction error decomposes into an error floor, a systematic error from an overly simple model, and a sensitivity-to-data error from an overly flexible one. It is the conceptual backbone for understanding underfitting, overfitting, and model complexity.

Error Decomposition

For a target $y=f(x)+\epsilon$ with $\mathbb{E}[\epsilon ]=0$, $\text{Var}(\epsilon )={\sigma }^2$, and estimator $\hat{f}$ trained on random data, the expected squared error at a point $x$ is:

$$\mathbb{E}[(y-\hat{f}(x){)}^2]=\underset{{\text{Bias}}^2}{\underbrace{(f(x)-\mathbb{E}[\hat{f}(x)]{)}^2}}+\underset{\text{Variance}}{\underbrace{\mathbb{E}[(\hat{f}(x)-\mathbb{E}[\hat{f}(x)]{)}^2]}}+\underset{\text{Irreducible}}{\underbrace{{\sigma }^2}}$$

• Bias: error from wrong assumptions; how far the average prediction is from the truth.
• Variance: how much predictions swing across different training sets.
• Irreducible error ${\sigma }^2$: noise floor no model can beat.

Underfitting vs Overfitting

Regime	Bias	Variance	Symptom
Underfitting	High	Low	High train and test error
Good fit	Balanced	Balanced	Low test error near train error
Overfitting	Low	High	Low train error, high test error

Diagnosis

A large train-to-test gap points to high variance (try more data or Regularization). Uniformly high error points to high bias (add capacity or features).

Model Complexity

• As complexity rises, bias falls and variance grows; total error is U-shaped.
• The optimal complexity minimizes bias${}^2$ + variance, not training error.
• More training data flattens the variance curve, allowing more complex models safely.
• Model capacity here is what VC Dimension and Generalization quantifies formally: a higher VC dimension permits lower bias but yields looser worst-case generalization guarantees.

Controlling the Tradeoff

• Regularization increases bias slightly to cut variance sharply (e.g. L2 shrinkage).
• Cross-Validation estimates test error to pick the complexity or penalty that balances the two.
• Ensembling (bagging) reduces variance; boosting reduces bias.