Backpropagation: How Networks Learn
A neural network makes a mistake. Which weights are to blame, and by how much? We turn one neuron into a computational graph, multiply slopes with the chain rule, send the blame backward through episode nine's XOR network, take one training step by hand, and train it until it solves XOR on its own. Plus: why backprop beats nudging every knob, PyTorch's autograd, and vanishing and exploding gradients. Every number on screen is real output from code you can run.
Chapters
- 0:00 Intro
- 0:46 Two different jobs
- 1:19 The computational graph
- 1:55 The chain rule
- 2:42 Episode four's slope, explained
- 3:17 Blame flows backward
- 3:59 One full training step
- 4:35 XOR learns itself
- 5:17 Why backprop is fast
- 5:58 Autograd
- 6:30 An old idea
- 7:03 What can go wrong
- 8:34 Try it yourself
- 9:07 Interview corner
- 10:16 Recap and next stop
Cheat sheet
- Backprop computes the gradient (every weight's slope); gradient descent uses it to take the step
- Chain rule: multiply the slopes along the path, from the loss back to the weight
- A weight's gradient = the blame at its neuron x the value flowing in (a zero input gets no blame)
- One backward pass costs a few forward passes; nudging needs a forward pass per weight
- PyTorch: loss.backward() fills .grad, and gradients add up, so zero them every step
- Vanishing gradients: sigmoid slopes (at most 0.25) multiply toward zero; fixes: ReLU, careful initialization (Glorot, He)
- Exploding gradients: blame grows layer by layer; fixes: gradient clipping, careful initialization
Code
Try it yourself (pip install torch)
import torch
x, y = 3.0, 1.0 # one input, and the right answer
w = torch.tensor(0.5, dtype=torch.float64, requires_grad=True) # the two knobs PyTorch will track
b = torch.tensor(-1.5, dtype=torch.float64, requires_grad=True)
def forward(w, b):
p = torch.sigmoid(w * x + b) # weighted sum plus bias, then the sigmoid
return p, -torch.log(p) # the prediction, and its log loss (the answer is 1)
p, loss = forward(w, b)
print(f"forward: p = {p.item():.4f}, loss = {loss.item():.4f}")
# By hand: multiply the slopes along the path, from the loss back to w.
p = p.item()
slopes = [-1 / p, p * (1 - p), x] # loss per p, p per z (sigmoid), z per w
print(f"by hand: {slopes[0]:.2f} x {slopes[1]:.2f} x {slopes[2]:.2f} = {slopes[0] * slopes[1] * slopes[2]:.4f}")
loss.backward() # PyTorch: every slope, in one backward pass
print(f"PyTorch: w.grad = {w.grad.item():.4f}, b.grad = {b.grad.item():.4f}")
nudge = 0.0001 # the check: nudge w, and watch the loss
with torch.no_grad():
change = forward(w + nudge, b)[1] - loss
print(f"nudge: {change.item() / nudge:.4f}")
forward(w, b)[1].backward() # a second backward, without zeroing first...
print(f"again: w.grad = {w.grad.item():.4f} (added up)")
w.grad.zero_() # ...so training loops zero the gradients every step
What we got: forward p = 0.5000 and loss 0.6931. By hand, -2.00 x 0.25 x 3.00 = -1.5000; PyTorch agrees (w.grad -1.5000, b.grad -0.5000), and the nudge test gives -1.4999. Call backward a second time without zeroing, and w.grad adds up to -3.0000.
backprop_check.pyvisual_data.py
Sources
- Rumelhart, Hinton & Williams, learning representations by back-propagating errors (1986)
- Linnainmaa, reverse-mode differentiation (1970 thesis; 1976 paper)
- Werbos, applications of advances in nonlinear sensitivity analysis (1982)
- Baydin et al., automatic differentiation in ML, a survey (2018)
- Bengio, Simard & Frasconi, learning long-term dependencies is difficult (1994)
- Glorot & Bengio, the difficulty of training deep feedforward networks (2010)
- Pascanu, Mikolov & Bengio, the difficulty of training recurrent networks (2013)
- Optical Recognition of Handwritten Digits, UCI (CC BY 4.0)
Transcript
Here's episode nine's XOR network, but with random weights. Give it one and zero, where the right answer is one, and it says point four two. It's wrong, and it has nine knobs. Which ones are to blame, and by how much? One algorithm answers that for every knob at once, by sending the error backward.
Repeat it five thousand times, and this network solves XOR, with weights nobody set by hand. The same idea trains models with billions of knobs. It's called backpropagation, and a classic interview question asks: what does it compute, and how is it different from gradient descent? Let's find out.
This is Just Known. From unknown, to known. Episode ten: backpropagation, how networks learn. Last episode, we set those XOR weights by hand. Real networks find their own, with episode three's walk downhill: move every knob a small step against its slope.
But that walk needs the slope of every knob. With two knobs, we worked them out on paper. A network has thousands, or billions, each buried behind layers of other knobs. Backpropagation is how you get them all. So keep the two jobs apart: backprop computes the slopes, and gradient descent uses them to take the step.
Mixing them up is a classic interview slip. Start with one neuron, drawn as a computational graph: every small step of the calculation is a node. Here, the input x is three, and the right answer is one. The knobs start at w equals point five, and b equals minus one point five. Multiply: one point five.
Add the bias: zero. The sigmoid turns that into p, a probability of point five. And the loss, from episode four, is minus the log of p: point six nine. That's the forward pass. Now we go backward.
Each node only needs to know its own slope: if my input moves a little, how many times as much do I move? The loss node's slope is minus one over p: minus two. The sigmoid's is p times one minus p: a quarter, its largest. And the multiply node's slope, for w, is just x: three. The chain rule says: multiply the slopes along the path.
Three, times a quarter, times minus two: minus one point five. Nudge w up by a thousandth, and z moves three thousandths, p a quarter of that, and the loss falls by one and a half thousandths. For b, the path skips the multiply: minus point five. That's backprop: this, for every knob, walking back from the loss. Notice: minus two times a quarter is minus a half, exactly p minus the right answer.
That's no accident. Through a sigmoid with log loss, those two slopes always cancel to prediction minus truth. So w's slope is prediction minus truth, times the input: episode four's formula, and now you know where it comes from. The minus sign means downhill is up: raise w. One gradient descent step, with a learning rate of point one, takes w to point six five, and the loss falls from point six nine to point four seven.
A network is just a bigger graph. Back to our random XOR network, with the input one, zero. The output says point four two, so its blame is prediction minus truth: minus point five eight. Now pass it back. Each weight into the output gets that blame, times the value coming in.
The first hidden neuron sent point seven, so its weight's gradient is minus point four one. Each hidden neuron gets the blame times its weight to the output, then times its own sigmoid slope: point zero five five for the first, point zero three for the second. And so on, back to the inputs. The second input was zero, so its two weights get no blame at all. They didn't contribute.
Training uses all four inputs: run four backward passes, and average them. That's a gradient for all nine knobs. Then comes the step from episode three. With a learning rate of one, every knob moves against its gradient. The output's bias, for example, had a gradient of minus point zero eight, so it rises from about point one two to point two.
Forward again, and the loss has dropped, from point seven zero seven to point six nine eight. That's one training step: forward, backward, update. And PyTorch gets the same nine gradients, down to rounding error. Now repeat that five thousand times. At first, almost nothing happens: for about five hundred steps, the loss sits on a plateau near point six nine, every output hovering around a half.
Then the network finds its way down. By step seven hundred eighty-two, it gets all four right. And it found its own solution. One hidden neuron switches off only for zero, one. The other, only for one, zero.
And the output fires when either is off. It's not the one we wrote by hand, but it works. It doesn't always work, though. From a hundred random starts, eighty-eight solve XOR. Eleven of the other twelve get stuck, with two answers right and two frozen at fifty-fifty.
Why not just nudge each knob, and see what the loss does? That works, and it's a great check. But it costs one extra forward pass per knob. Take a network the size of episode nine's digit reader, with one thousand two hundred ten knobs. Nudging them all takes one thousand two hundred eleven forward passes: on our computer, over five hundred times slower than backprop.
For a network of three hundred thousand knobs, we estimate over an hour per step, against about forty milliseconds. Backprop's one backward pass costs only a few forward passes, however many knobs there are. With a billion knobs, the nudge test needs a billion forward passes. Backprop still needs one forward, and one backward. In practice, you never write this yourself.
PyTorch's autograd records every operation as your code runs, building the graph on the fly. Call loss dot backward, and it walks that graph in reverse, multiplying slopes, and leaves each knob's gradient attached to it. One catch: those gradients add up. Call backward again without clearing them, and w's gradient doubles, to minus three. That's why every PyTorch training loop zeroes the gradients before each step.
Forgetting it is a classic bug, and a classic interview question. The idea is older than deep learning. The chain rule goes back to Leibniz, in sixteen seventy-six. In nineteen seventy, a Finnish master's student, Seppo Linnainmaa, described the efficient backward method for any calculation, now called reverse-mode automatic differentiation. Paul Werbos applied it to neural networks in nineteen eighty-two.
And in nineteen eighty-six, David Rumelhart, Geoffrey Hinton and Ronald Williams showed in Nature that it lets hidden layers learn useful features of their own, and made it famous. So what can go wrong? Blame can fade away. At every layer, blame is multiplied by a weight and by the neuron's slope, and a sigmoid's slope is never more than a quarter. Here's episode nine's ten-layer sigmoid network, at its start.
Each layer passes back only about a ninth of the blame it receives. By the first layer, less than a billionth of the output's blame is left. Those early knobs barely move, and that's why the network was stuck at fifty percent. This is the vanishing gradient problem, analyzed by Sepp Hochreiter in nineteen ninety-one, and by Yoshua Bengio and colleagues in nineteen ninety-four. A ReLU's slope is one when it's active, so with ReLU and well-sized starting weights, the blame stays roughly level.
The opposite can happen too. Start that same ReLU network with every weight twice as big, and each layer passes back twice as much blame. By the first layer, it's about a thousand times bigger than before, and the starting loss is eighty-seven. Steps that big overshoot, like episode three's too-high learning rate. The fixes.
Use ReLU, but watch its flat side: a dead ReLU has a slope of zero, so no blame gets through. Our little XOR network, with ReLU, got stuck from seventy-six of a hundred starts, mostly with a dead neuron. Start the weights at the right size, as papers in twenty ten and twenty fifteen worked out. Clip the gradient when it grows too big. And skip connections and normalization, coming in episodes eleven and twelve, help too.
You can run this yourself, in about twenty lines of PyTorch. The code is in the description. It builds our one-neuron graph and gets the gradient three ways. By hand, multiplying the slopes: minus one point five. PyTorch's loss dot backward: minus one point five for w, and minus point five for b.
And the nudge test: minus one point four nine nine nine, the same, up to the size of the nudge. Then it calls backward again without zeroing, and the gradient doubles, to minus three. Interview corner. What does backpropagation compute, and how is it different from gradient descent? And why can gradients vanish in a deep network?
Pause here, and try answering in thirty seconds.
Here's a strong answer. Backpropagation computes the gradient of the loss with respect to every weight and bias. It runs forward, then applies the chain rule backward, layer by layer, reusing what it has already computed. That gets every gradient for the cost of a few forward passes, where nudging each weight would need one forward pass per weight. Gradient descent, or another optimizer, then uses those gradients to update the weights.
In PyTorch: zero the gradients, call backward, then step. Gradients vanish because backprop multiplies by every layer's weights and activation slopes, and a sigmoid's slope is at most a quarter. ReLU, careful starting weights, and later, residual connections and normalization, keep the signal flowing. To stand out, mention checking gradients with a nudge test, and that backprop is a special case of reverse-mode automatic differentiation. Let's recap.
A network is a computational graph. Backprop sends the blame backward, multiplying by each local slope, and each weight's gradient is the blame at its end, times the value at its start. One backward pass gets every gradient, and gradient descent takes the step. Repeat, and XOR learns itself. But in deep stacks, blame can vanish, or explode.
We trained on just four examples. Real networks learn from millions, in small batches, with smarter steps, and graphics cards to do the math. That's episode eleven: training deep networks, and why GPUs. This is Just Known. From unknown, to known.
Note: the neuron, XOR inputs and two rings are made-up data; the handwritten digits are real. Timings are from our CPU; the large network's nudge time is extrapolated (labeled). Pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.
