AI · 05 Artificial Intelligence

Overfitting, Bias–Variance & Regularization

Overfitting, Bias–Variance & Regularization

A model that scores 100% on its training data and 50% on new data: that's overfitting. We fit curves of every degree from 1 to 15 to made-up delivery orders and watch training error fall to zero while validation error explodes. Then: train / validation / test, the bias-variance trade-off, ridge (L2) and lasso (L1) regularization, cross-validation, more data vs a simpler model, and data leakage, the bug that turns pure noise into 79.5% accuracy. Every number on screen is real output from code you can run.

Chapters

Cheat sheet

Code

Try it yourself (pip install numpy)
import numpy as np
from numpy.polynomial import Polynomial

# 40 made-up orders. Hidden rule: a lunch rush and a dinner rush, plus random noise.
rng = np.random.default_rng(0)
hour = rng.uniform(11, 23, size=40)
minutes = 30 + 10 * np.sin(hour + 0.6) + rng.normal(0, 3, size=40)
train, val, test = np.split(rng.permutation(40), [16, 28]) # 16 to learn from, 12 to choose with, 12 locked away

def error(curve, rows): # typical miss in minutes (root mean squared error)
return np.sqrt(np.mean((curve(hour[rows]) - minutes[rows]) ** 2))

val_errors = {}
for degree in range(1, 16):
curve = Polynomial.fit(hour[train], minutes[train], degree) # learn from the training orders only
val_errors[degree] = error(curve, val)
print(f"degree {degree:2d} train {error(curve, train):5.2f} validation {val_errors[degree]:13,.1f}")

best = min(val_errors, key=val_errors.get) # choose with the validation set
final = Polynomial.fit(hour[train], minutes[train], best)
print(f"best on validation: degree {best}")
print(f"test error, opened once: {error(final, test):.1f} minutes")

What we got: degree 1 misses by 7.9 minutes on validation (underfitting), degree 6 by 3.3 (the best; 4.2 on the test set), and degree 15 hits every training order (0.00) but misses validation by 4,965,042.9 minutes.

Sources

Transcript

Last episode, our spam filter scored ninety-six point eight percent on messages it had never seen. Here's a model that does better: a hundred percent, on its training data. Every single example, right. Now show it a hundred new examples. Fifty percent.

A coin flip. The twist: this data was pure noise. Five thousand columns of random numbers, and a coin-flip label. There was nothing to learn, and the model still scored perfectly, because it memorized the answers. That's overfitting.

A perfect training score is a warning sign, not a trophy. And it leads to a classic interview question: what is the bias-variance trade-off? Let's find out.

This is Just Known. From unknown, to known. Episode five: overfitting, bias and variance, and regularization. Back to our delivery app, with a new question: how long does a delivery take, depending on when the order is placed? We made up forty orders from a hidden rule: a lunch rush around half past one, a dinner rush around ten to eight, and random delays on top.

The model never sees that rule. Then we split the orders. Sixteen to learn from: the training set. Twelve to compare models: the validation set. And twelve locked away until the very end: the test set.

In episode three, our model was a straight line. To bend it, add columns: the hour squared, the hour cubed, and so on. That's polynomial regression. It's still linear regression underneath, and the highest power is called the degree. Degree fifteen has sixteen knobs, one for each training order.

Let's fit three, and measure the error: the typical miss, in minutes. Degree one is nearly flat, and sails through both rushes: seven point two minutes. Degree six bends with the rushes: two point eight. And degree fifteen passes exactly through every training order: zero point zero zero. Judged on its training data, degree fifteen wins easily.

Now the twelve validation orders, which no model has seen. Degree one misses by seven point nine minutes. Degree six, three point three. And degree fifteen misses by almost five million minutes. Take one order at three eighteen in the afternoon.

It actually took thirty-four minutes. Degree fifteen predicts about minus thirty-eight hundred. Degree one is underfitting: too simple to capture the pattern, so it's wrong everywhere. Degree fifteen is overfitting: it memorized the noise in its training data, so it fails on anything new. Now sweep every degree from one to fifteen.

This is the picture to remember. Training error only ever goes down: every extra knob lets the curve hug the training orders a little closer. Validation error falls at first, bottoms out at degree six, then climbs, and from degree ten it shoots off the chart. On the left, both errors are high: underfitting. On the right, training keeps improving while validation gets worse: overfitting.

That widening gap is your warning sign, and the sweet spot sits in between. So why lock away a third set? Because choosing is a kind of training. We tried fifteen models and kept the one that did best on validation. Part of that win is luck, so its validation score flatters it.

The test set is the final exam: opened once, at the end, for the model you've already chosen. Degree six missed by three point three minutes on validation, and by four point two on the test set. A bit worse, which is normal. Keep tuning until the test score looks good, and it stops being a test. Why does this happen?

Imagine replaying the same day: the same sixteen order times, but new random delays. Fit each model again, eight times. Degree one gives almost the same flat line every time, always wrong in the same way. That's bias: error from a model that's too simple for the pattern. Degree fifteen gives a wildly different curve every time, because it chases the random delays.

That's variance: how much the model changes with the luck of the data. Degree six stays close to the hidden rule, every time. For squared error, this splits into one equation. The expected error on a new order is bias squared, plus variance, plus noise. The noise is the random delay itself, and no model can beat it.

As the degree goes up, bias falls and variance rises. Degree one: bias squared forty-six, variance about one. Degree six: about one and a half, and under four, the lowest total. Degree fifteen: almost no bias, and a variance in the tens of billions. That's the bias-variance trade-off.

A 1992 paper by Geman, Bienenstock and Doursat called it the bias-variance dilemma, and made it famous in machine learning. So how do you keep a flexible model from going wild? Regularization: add a penalty for big weights to the loss. To bend through every point, our degree-fifteen curve needed weights over a billion. Ridge regression, or L two, adds alpha times the sum of the squared weights, so big weights cost something.

At a small alpha, the largest weight drops to about fifty, and the validation error falls from millions of minutes to under five. Push alpha too far, and the curve flattens: underfitting again. Ridge regression goes back to Hoerl and Kennard, in 1970. And our spam filter already used this idea: scikit-learn's logistic regression adds an L two penalty by default. Swap the squares for absolute values, and you get L one regularization, called the lasso, from Robert Tibshirani in 1996.

Ridge shrinks every weight, but almost never all the way to zero. The lasso pushes some weights to exactly zero. At its best setting here, it switched off nine of the fifteen weights, kept six powers of the hour, and scored three point nine on validation. So L one also picks features for you. L two just shrinks them.

One more catch: our validation set is only twelve orders. A different twelve, and we might crown a different winner. Cross-validation fixes that. Pool the training and validation orders, cut them into five folds, and take turns: fit on four, score on the fifth. Then average.

Degree four shows why: two point two minutes on one fold, nine on another. Degree six stayed between three and about four on every fold, an average of three point four five. Degree fifteen averaged over twelve hundred. Cross-validation picks degree six too, with the test set still locked. A classic 1974 paper by Mervyn Stone helped formalize the method.

Another cure for variance is more data. Keep collecting orders from the same hidden rule, and score each model on a thousand fresh ones. Trained on sixteen orders, degree fifteen misses by over a million minutes. On thirty, by eleven point six. On a hundred, three point two.

And on a thousand, it edges out degree six, right at the noise floor of three. But degree one stays around seven and a half minutes, no matter how much data it gets. More data cures variance, not bias. So when data is scarce, use a simpler or regularized model. When it's plentiful, you can afford a more flexible one.

So what can go wrong? The sneakiest failure is data leakage: information from the validation or test data slipping into training. Back to the coin-flip noise. First, pick the twenty columns that best match the labels, using all two hundred rows. Then cross-validate.

Seventy-nine point five percent, on pure noise! The selection had already peeked at every fold's answers. Put the selection inside a pipeline, so each fold picks its columns from its own training rows, and the score drops to fifty-one percent. A coin flip, which is the truth. Leaks hide everywhere: scaling with statistics from all the data, duplicate rows on both sides of the split, or a forecasting model that trains on the future.

The rule: anything that learns from data learns from the training rows only. You can run this sweep yourself, in about twenty lines of NumPy. The code is in the description. It makes up the forty orders, splits them sixteen, twelve, and twelve, and fits every degree on the training orders only. Watch the training column fall all the way to zero, while the validation column bottoms out at degree six, then explodes.

Then it opens the test set, once: four point two minutes. A second script runs the coin-flip experiment: a hundred percent on training, fifty on new rows, and the leak that turns noise into seventy-nine point five. Interview corner. What is the bias-variance trade-off? And how can you tell whether a model is overfitting or underfitting?

Pause here, and try answering in thirty seconds.

Here's a strong answer. Bias is error from assumptions that are too simple: the model misses the real pattern, which is underfitting. Variance is how much the model changes with its particular training data: it fits the noise, which is overfitting. For squared error, the expected error is bias squared, plus variance, plus irreducible noise. More flexibility lowers bias but raises variance, so you look for the sweet spot.

To diagnose, compare training and validation error. Both high: underfitting. Low training error, but much higher validation error: overfitting. Underfitting needs more flexibility or better features. Overfitting needs more data, L one or L two regularization, or a simpler model, with the setting chosen by cross-validation.

To stand out, add two things. Keep a test set you open only once, because a validation score used for tuning is optimistic. And in very large neural networks, test error can fall again after they fit the training data perfectly, a pattern called double descent. Let's recap. Training error always rewards complexity, so judge models on data they haven't seen.

Too simple means high bias: underfitting. Too flexible means high variance: overfitting. Regularization, cross-validation, more data, or a simpler model keep you in between. And guard your test set: anything that learns from data learns from the training rows only. All episode, we've judged models by a single number.

But remember episode four's lazy spam filter: eighty-six point six percent accurate, and zero spam caught. One number can hide a lot. Next, we'll find out how to tell if your model is actually any good. That's episode six. This is Just Known.

From unknown, to known.

Note: the delivery orders and the noise lab are made-up data from stated rules; every number on screen is real output from the code. Pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.