AI · 03 Artificial Intelligence

How Machines Learn: Linear Regression & Gradient Descent

How Machines Learn: Linear Regression & Gradient Descent

How does a machine actually learn? It walks downhill, blindfolded. We take the delivery-time model from episode two and let it find its own knob settings: a loss that measures how wrong it is (mean squared error), the loss landscape, the gradient as the slope under your feet, and gradient descent stepping downhill. Then the learning rate (too small crawls, too big explodes), reading a loss curve, feature scaling, batch vs stochastic vs mini-batch, why we don't just solve it, and what one outlier does to your line. All with real numbers from code you can run.

Chapters

Cheat sheet

Code

Try it yourself (Python + NumPy)
pip install numpy scikit-learn

import numpy as np
from sklearn.linear_model import LinearRegression

# The same 10 made-up deliveries as episode 2 (hidden rule: 8 minutes + 3 per km, plus noise)
rng = np.random.default_rng(1)
km = rng.uniform(1, 10, size=10).round(1)
minutes = (8 + 3 * km + rng.normal(0, 3, size=10)).round()

w, b = 0.0, 0.0 # the two knobs, starting anywhere
lr = 0.01 # learning rate: how big a step to take

for step in range(5001):
err = w * km + b - minutes
loss = np.mean(err ** 2) # mean squared error
if step in (0, 1, 10, 100, 1000, 5000):
print(f"step {step:4d} loss {loss:8.3f} w {w:6.3f} b {b:6.3f}")
w -= lr * 2 * np.mean(err * km) # step downhill along w
b -= lr * 2 * np.mean(err) # ...and along b

print(f"gradient descent: minutes = {w:.2f} x km + {b:.2f}")
check = LinearRegression().fit(km.reshape(-1, 1), minutes)
print(f"scikit-learn: minutes = {check.coef_[0]:.2f} x km + {check.intercept_:.2f}")

What we got: loss 661.4 at step 0, 37.7 after one step, 1.373 after 5,000, landing on 3.17 per km + 6.52, the same line as scikit-learn. Now try lr = 0.001 (it crawls) and lr = 0.03 (it blows up).

Sources

Transcript

Last episode, we found the best settings for our delivery model by turning its two knobs by hand and watching the misses shrink. That worked because there were only two knobs. GPT-3 has 175 billion. Even ten knobs, with just a hundred settings each, make a hundred billion billion combinations. You could never try them all.

So how does a machine find the right settings? It walks downhill, blindfolded. That idea is called gradient descent, and it trains almost every model you've heard of, from a straight line to ChatGPT. It's also one of the most common machine learning interview questions: explain gradient descent. Let's take it one step at a time.

This is Just Known. From unknown, to known. Episode three: how machines learn. Linear regression, and gradient descent. We're back with our ten made-up deliveries, and our model with two knobs: minutes equals w times the distance, plus b.

Finding the straight line that best fits the data is called linear regression. It's the oldest trick in machine learning, and the perfect place to see how learning works. But first, we need to say exactly what best means. To improve, the model needs one number that says how wrong it is. That number is called the loss.

Here's the most common one. Take every miss, square it, and average them. That's the mean squared error. Why square? Squares are never negative, so misses above and below the line can't cancel out.

Big misses count much more than small ones. And the result is smooth, which matters in a minute. With both knobs at zero, the loss is 661. At the best settings, it's 1.37. Take the square root, and that's a typical miss of about a minute.

Fitting lines this way is more than two centuries old. Legendre published the method of least squares in 1805, and Gauss in 1809. Now, picture every possible setting of the two knobs as a point on a map: w across, b up. At each point, raise the ground to that setting's loss. You get a landscape, and training means getting to its lowest point.

For linear regression like ours, the landscape is a smooth bowl with exactly one bottom. That's the best line. With two knobs, we can draw the whole bowl. With a billion knobs, nobody can. You're standing somewhere in the fog.

So, blindfolded, how would you get down a hill? Feel the ground under your feet. Which way does it tilt? Take a small step downhill. Feel again.

Repeat. That's gradient descent. The slope under your feet is the gradient. For each knob, it answers one question: if I nudge this knob up a little, how fast does the loss go up? The gradient points uphill, the steepest way up.

So you step the opposite way. The method is usually credited to Augustin-Louis Cauchy, in 1847. How a network with billions of knobs computes all those slopes at once is episode ten. Here's the whole rule, for every knob: the new value is the old value, minus the learning rate times the slope. For our model, the two slopes are simple.

For w, it's twice the average of each miss times its distance. For b, it's twice the average miss. Read them like this. If the predictions are too high, the misses are positive, so both knobs go down. Too low, and they go up.

Let's take the first step, with real numbers. Both knobs start at zero, so every prediction is zero, and every miss is negative. The slope for w comes out at minus 317, and for b, minus 48. A negative slope means downhill is up, so both knobs go up: times the learning rate of 0.01, w rises by 3.17, and b by 0.48. One step, and the loss drops from 661 to under 38.

Keep going. After ten steps, the loss is down to 7.6. W is already close, but b is lagging behind. The valley is long and narrow: steep across, almost flat along its floor. So the ball crosses quickly, then crawls.

After a hundred steps, the loss is 4.6. After a thousand, 1.38. And after five thousand steps: 3.17 minutes per kilometer, plus 6.52. Exactly what scikit-learn found in episode two. The learning rate decides how big each step is, and it's the most important setting you'll choose yourself.

Settings like that, which you pick rather than the model learning them, are called hyperparameters. Too small, and learning crawls. At 0.001, after a hundred steps the loss is still 7.6, and a thousand steps later it still hasn't arrived. Too big, and every step overshoots the valley and lands higher up the other side. At 0.03, the loss goes from 661 to 1,219 after one step, to 312 thousand after ten, and then to numbers too big to say.

That's called diverging. For this data, anything above about 0.025 blows up. Picking learning rates well, and changing them during training, comes back in episode eleven. In a real model you can't see the landscape. But you can always watch one thing: the loss, step by step.

Plot it, and you get a loss curve, the first thing to check in any training run. A healthy curve drops fast, then flattens as it reaches the bottom. A curve that's still sloping down when training ends means the learning rate is too small, or you stopped too early. And a curve that shoots up means it's diverging, so lower the learning rate. Here are our three runs on one chart, and you can read all three at a glance.

Why was b so slow? Our distances run from one to ten kilometers, so the loss reacts far more strongly to w than to b. The valley is stretched: about two hundred times steeper across than along its floor. A common fix is to rescale the input, so it's centered on zero with a spread of one. The valley turns round, and you can safely take bigger steps.

With raw kilometers, it took 822 steps to get within one percent of the best loss. Rescaled, 25. So far, every step looked at all ten deliveries before moving. That's batch gradient descent. With a billion examples, a single step would take forever.

Stochastic gradient descent looks at one example per step. Each slope is a rough guess, so the path wobbles. In our run, the loss even went up after five passes through the data. But after four hundred passes, it was at 1.38, right near the bottom. One pass through all the data is called an epoch.

Deep learning uses the middle ground: mini-batches, a small random chunk of examples per step, often a few dozen to a few hundred. The stochastic idea grew out of work by Robbins and Monro, in 1951. Here's a fair question. For a straight line, you don't have to walk: there's an exact solution you can compute directly, which jumps straight to the bottom. That's what scikit-learn's linear regression does, with no gradient descent at all.

So why walk? Because for neural networks, there's no such formula. Their landscapes are bumpy, with millions or billions of dimensions, and the only way down is step by step. Every knob in a model like ChatGPT was set by a version of what you just saw. GPT-3, for example, was trained with Adam: gradient descent with a smarter, adaptive step size.

Neural networks start in episode nine. So what can go wrong? We've seen the learning rate. Here's the other classic: outliers. Add one more delivery: two kilometers, sixty minutes, stuck behind an accident.

Squared error punishes big misses hard, so the line bends toward that one point. It becomes 1.38 minutes per kilometer, plus 20. On the ten normal deliveries, it's now off by more than five minutes on average. A robust loss, called the Huber loss, treats big misses more gently. With it, the line stays at 3.13 per kilometer, plus 6.89, off by under a minute.

One bad row can drag your whole model. Look at your data before you trust your line. You can run this yourself, in about fifteen lines of NumPy. The code is in the description. Two knobs start at zero.

Every step computes the predictions, the misses, and the loss. Then the slopes, and one small step downhill for each knob. Run it, and you can watch the loss fall: 661, then 38, then 9.4, all the way down to 1.37. It lands on exactly the same line as scikit-learn. Then change one number, the learning rate, and watch it crawl at 0.001, or blow up at 0.03.

Interview corner. Explain gradient descent. And what happens if the learning rate is too high? Pause here, and try answering in thirty seconds.

Here's a strong answer. Gradient descent is how we train a model by minimizing a loss function, a number that measures how wrong the model is. At each step, we compute the gradient of the loss with respect to every parameter. The gradient points uphill, so we move each parameter a small step in the opposite direction, scaled by the learning rate. We repeat until the loss stops improving.

If the learning rate is too high, the steps overshoot the minimum, and the loss can bounce around or blow up. Too low, and training is painfully slow. In practice we use mini-batches: a small random sample of examples per step. To stand out, mention two things: scaling your features makes the landscape rounder and training faster, and neural networks have no closed-form solution, which is why everything is trained this way. Let's recap.

A loss turns "how wrong is the model" into one number, and mean squared error is the classic. Gradient descent finds good settings without seeing the whole landscape: feel the slope, step downhill, repeat. And the learning rate decides whether you crawl, land, or blow up. So far, our model predicts a number: minutes. But many questions are yes or no.

Is this email spam? Will this customer cancel? That needs a new trick: turning a line into a probability. That's episode four. This is Just Known.

From unknown, to known.

Note: the delivery data is made up from a hidden rule (as episode 2 says); every number on screen is real output from the code above. The loss landscape is computed from that data; pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.