AI · 11 Artificial Intelligence

Training Deep Networks (and Why GPUs)

Training Deep Networks (and Why GPUs)

A real network learns handwritten digits in mini-batches, and one changed number turns its loss into NaN. Epochs and batches, momentum and Adam, learning-rate warmup and decay, batch norm, dropout, early stopping, debugging a NaN, and why GPUs win (and when they lose), timed on a laptop. Every number on screen is real output from code you can run.

Chapters

Cheat sheet

Code

Try it yourself (pip install torch scikit-learn)
import copy
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True) # 8 x 8 handwritten digits, bundled with scikit-learn
X_train, X_test, y_train, y_test = train_test_split(X / 16, y, test_size=0.25, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
X_train, X_val, X_test = (torch.tensor(a, dtype=torch.float32) for a in (X_train, X_val, X_test))
y_train, y_val, y_test = (torch.tensor(a) for a in (y_train, y_val, y_test))

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(64, 512), nn.ReLU(), nn.Dropout(0.2),
nn.Linear(512, 512), nn.ReLU(), nn.Dropout(0.2), nn.Linear(512, 10))
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.CrossEntropyLoss()
print(f"{len(X_train)} training images, {sum(p.numel() for p in model.parameters()):,} knobs")

scores = []
for epoch in range(1, 21): # an epoch: one pass over the training images
model.train() # dropout on
total = 0.0
for batch in torch.randperm(len(X_train)).split(32): # shuffled mini-batches of 32
optimizer.zero_grad()
loss = loss_fn(model(X_train[batch]), y_train[batch])
loss.backward()
optimizer.step() # one step per mini-batch
total += loss.item() * len(batch)
model.eval() # dropout off
with torch.no_grad():
scores.append((model(X_val).argmax(1) == y_val).float().mean().item())
if scores[-1] == max(scores): # best epoch so far: keep its weights
best, best_epoch = copy.deepcopy(model.state_dict()), epoch
if epoch % 5 == 0:
print(f"epoch {epoch}: training loss {total / len(X_train):.4f}, validation accuracy {scores[-1]:.1%}")

model.load_state_dict(best) # early stopping: back to the best epoch
with torch.no_grad():
test_acc = (model(X_test).argmax(1) == y_test).float().mean().item()
print(f"best epoch: {best_epoch} (validation {max(scores):.1%}); test accuracy: {test_acc:.1%}")

What we got: 301,066 knobs; validation 96.7% at epoch 5, best 98.9% at epoch 18; test accuracy 97.3%.

Sources

Transcript

Last episode, a tiny network learned from four examples. Today, a real one: episode ten's digit reader, with three hundred thousand knobs, learning from about a thousand handwritten digits, thirty-two at a time. With a learning rate of point one, the loss falls, step by step. Now change that one number, to five. The loss climbs, then explodes, and at step seventeen, it isn't a number at all.

It's NaN. That's a classic interview question: your training loss just turned into NaN. What happened, and what do you check first? Let's find out, and see why all this runs on graphics cards.

This is Just Known. From unknown, to known. Episode eleven: training deep networks, and why GPUs. Real training doesn't look at every example before each step. It shuffles the data, cuts it into mini-batches, and takes one step per batch.

Our one thousand seventy-seven training images, in batches of thirty-two, make thirty-four steps. One full pass is an epoch, and each step is called an iteration. So ten epochs here are three hundred forty steps. Every step is episode ten's loop: zero the gradients, a forward pass, loss dot backward, and the optimizer's step. How big should a batch be?

Here's plain gradient descent, three ways, ten epochs each. The whole training set per step: each step is exact, but there's only one per epoch, and the loss barely moves: two point three one to two point two two. One image per step: a thousand seventy-seven noisy steps per epoch, each pointing only loosely downhill. The loss jumps around, even back up to two point oh eight. Thirty-two per step sits in between: the loss falls to point one eight.

And the computer handles a batch in one go: an epoch took seventy-three milliseconds on our CPU, against eight hundred thirty-five, one image at a time. Next, smarter steps. Remember episode three's long, narrow valley? Steep across, almost flat along the floor. Here's a made-up one, a hundred times more curved across than along.

Plain gradient descent bounces from wall to wall, and crawls along the floor. Push its learning rate past point five, and it bounces right out. Even at its best setting, it takes a hundred thirty-four steps to settle. Momentum adds each new slope to nine-tenths of the last step, like a heavy ball rolling. The swings fade, the speed builds along the floor, and it settles in sixty-three steps.

Boris Polyak described it in nineteen sixty-four. Adam goes further, giving every knob its own step size. It keeps a running average of each knob's slope, like momentum, and of how big that slope has been, then divides one by the other. So steep and gentle knobs take similar steps, and it settles in fifty-six. On the digits, at the same learning rate, momentum leaves plain gradient descent far behind, and Adam, at its usual setting, is fastest of all.

It's from twenty fourteen, and a common default. But tune each one's learning rate, and all three land within a point. On some tasks, well-tuned SGD with momentum even generalizes better. All of them still need a learning rate, often called the single most important setting. Episode three showed why: too small crawls, too big diverges.

Here it is for real. At point zero one, the loss crawls. At point one, it falls nicely. At two, the network breaks, and ends no better than guessing. At five, the loss passes a hundred by step nine, ten billion by step eleven, and at step seventeen, it's NaN.

Same network, same data, same start. One number changed. So real training changes the learning rate as it goes, as episode three promised. A popular recipe: warm up, then decay. Take Adam at a learning rate of point one, a hundred times its usual setting.

Held constant, its first step throws the loss from two point three to a hundred forty-six, and twenty epochs later, it's stuck at sixty-four percent. Decay alone, along a cosine curve, still spikes, and reaches ninety-one. Now warm up: raise the rate in a straight line over the first epoch, then decay. No big spike, and ninety-seven percent. Warmup took off in twenty seventeen, with large-batch training and the Transformer.

Deep networks also need a healthy signal through every layer. Episode ten covered the starting weights' size. They must also be random: give every weight and bias the same value, and all five hundred twelve neurons in a layer get the same gradient, stay identical, and learn nothing. Episode nine's ten-layer sigmoid network was stuck at fifty percent, with less than a billionth of the blame reaching its first layer. Add batch normalization: after each layer's weighted sum, rescale every neuron's values across the batch to a standard size.

Now the first layer gets four times the output's blame, and by pass fifty, it's ninety-nine percent right. Layer normalization does this within each example, and it's the kind inside transformers. A network this big can also memorize, episode five's overfitting. Train it on just two hundred images, and the training loss heads to zero, but the validation loss bottoms out at epoch twenty-four, then climbs back up. The simplest fix is early stopping: keep the weights from the best validation epoch, the guard boosting used in episode seven.

Another is dropout: on every training step, each hidden neuron is switched off with a fifty-fifty chance, so none can lean on any other. Here, the validation loss bottoms out lower, and later, from all three starts we tried. But dropout is only for training: call model dot eval, and it's off. Now, the hardware. Almost all of this is matrix multiplication.

A batch is a tensor, a grid of numbers, thirty-two images by sixty-four pixels, and each layer multiplies it by a grid of weights: nine point six million multiply-adds per batch, and about twice that going backward. A CPU has a few powerful cores; ours has sixteen. A graphics card has thousands of simple ones: NVIDIA lists three thousand seventy-two on our laptop's GPU, fed by much faster memory. So we timed one multiply: two grids, eight thousand one hundred ninety-two on a side, over half a trillion multiply-adds. Sixteen CPU cores took two point nine seconds.

The GPU took under a fifth of a second, about fifteen times faster. But at small sizes, the GPU loses: every operation has a fixed start-up cost, and data has to cross over to the card first. For a sixty-four by sixty-four multiply, our CPU was about three times faster. From a hundred twenty-eight on a side, the GPU pulls ahead. Training is the same.

An epoch of episode nine's little digit reader took thirty-nine milliseconds on the GPU, and seventeen on a single CPU core. Our three-hundred-thousand-knob network ran one and a half times faster there, and a wide one, with seventeen million knobs, fourteen times faster. In two thousand nine, a Stanford team trained networks up to seventy times faster on graphics cards, and in twenty twelve, AlexNet trained on two, as we'll see in episode twelve. Back to our NaN. At a learning rate of five, every step overshot, and the weights grew, to numbers thirty-four digits long by step seventeen.

Then the network's scores passed the largest number a thirty-two-bit float can hold, about three point four times ten to the thirty-eighth, and became infinity. To turn scores into a loss, softmax first subtracts each image's largest score. Infinity minus infinity is not a number: NaN. And NaN spreads. One backward pass later, forty-five thousand gradients were NaN, and a step later, nearly nine in ten weights.

So how do you debug it? Log the loss and gradient size every step, to catch the climb before the crash. PyTorch's anomaly detection names the operation: here, the log softmax. Then lower the learning rate: at point one, this run is fine. Warmup alone only delayed the NaN, and clipping stopped it, but nothing was learned.

And check for a log of zero, or bad data. At scale, it's expensive. When Meta's hundred-seventy-five-billion-parameter model diverged in training, on nine hundred ninety-two GPUs, the team lowered the learning rate and restarted from an earlier checkpoint. Google's PaLM hit about twenty loss spikes. So big runs save checkpoints.

You can run this yourself: under forty lines of PyTorch, in under a minute on a laptop CPU. The code is in the description. It trains our three-hundred-thousand-knob network on the digits for twenty epochs: shuffled batches of thirty-two, Adam, and dropout, switched on by model dot train and off by model dot eval. The best validation score, ninety-eight point nine percent, came at epoch eighteen, so it keeps those weights, and scores ninety-seven point three percent on the test set. A second script times your CPU against your GPU.

Interview corner. Your training loss suddenly turns into NaN. What are the likely causes, and how do you debug it? And why do we train on GPUs? Pause here, and try answering in thirty seconds.

Here's a strong answer. A NaN usually means something overflowed to infinity, or zero was divided by zero. Most often, the learning rate is too high: the loss and weights blow up until infinity minus infinity gives NaN. Other causes: a log of zero, bad data, or sixteen-bit overflow. To debug, log the loss and gradient norm every step, find the first bad step, and check its batch.

Then lower the learning rate, add warmup or clipping, and use losses that take raw scores, which are numerically stable. GPUs help because training is mostly big matrix multiplies, run in parallel on thousands of cores with fast memory. For tiny models, launch and transfer overhead can make them slower. To stand out, add that clipping can hide a NaN without fixing its cause. Let's recap.

Training shuffles the data into mini-batches: one step per batch, one epoch per pass. Momentum and Adam take smarter steps, and the learning rate, warmed up and decayed, still matters most. Normalization keeps the signal flowing, dropout and early stopping fight memorizing, and too big a step ends in NaN. Under it all are matrix multiplies, which is why graphics cards win, once the work is big enough. So far, our networks have read each image as a flat list of sixty-four numbers.

Next, they learn to see shapes. That's episode twelve: how machines see, with convolutional networks. This is Just Known. From unknown, to known.

Note: the valley and the two rings are made-up; the digits are real. Times are from our laptop (RTX 4060 Laptop GPU, Core i7-13700HX) and vary run to run. Pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.