Supervised Learning and the Bias-variance Tradeoff

What this topic is

Supervised learning is fitting a function from labelled examples so it predicts the label on rows you have never seen. Its error at any point splits into exactly three pieces: bias, the error left because your model class is too rigid to represent the truth; variance, the error caused by the fit moving around when the training sample changes; and irreducible noise, the part of the outcome nothing could ever have predicted. The bias-variance tradeoff is the observation that, over a family of models, the first two move in opposite directions as you add complexity.

Why it matters

Almost every argument about a predictive model in a business is an argument about complexity conducted without vocabulary. One side says the model is too simple; the other says it is overfitted; nobody can settle it because nobody has decomposed the error. Once you have, the argument becomes an arithmetic problem with an answer, and — much more usefully — it tells you where the next budget goes. A variance-dominated model is fixed by more data or more regularization, both cheap. A bias-dominated model is not fixed by more data at all, and a team that buys another year of history to fix one has bought nothing.

Who should learn it

Operators commissioning or reviewing a predictive model, analysts building one, and anyone being asked to approve spending on data acquisition to make a model better.

What you will understand

• How to split a model's expected squared error into bias, variance, and noise, with the arithmetic worked through
• The learning-curve diagnostic that tells you whether more data will help before you buy any
• Why the same decomposition does not hold cleanly for classification, and what survives anyway
• Where the classical U-shaped curve stops describing modern practice, and what to do about it

Prerequisites

• Statistical Significance and What a P-value Does not Mean
• Multi-armed Bandits and the Explore-exploit Tradeoff

Common misconception

"The model is 91% accurate, so it is a good model." Accuracy on which rows, and compared to what? A model that predicts nobody churns is 94% accurate in a book with 6% churn. And a single accuracy number tells you nothing about the thing you actually need to decide: whether the error you are seeing would shrink if you fed the model more history. Two models with identical holdout accuracy can be at opposite ends of the tradeoff — one rigid and stable, one flexible and jittery — and the correct next investment for them is the opposite in each case.