Neurons & Neural Networks
Remove one tiny function from a 10-layer neural network, and it drops from 100% to 47%: no better than a coin flip. We build up from a single artificial neuron (it's exactly logistic regression from episode four), the bias, Rosenblatt's perceptron and the XOR problem it can't solve, two layers that solve it, activation functions (sigmoid, tanh, ReLU), the forward pass of a real network on handwritten digits, why stacked layers collapse into one line without activations, universal approximation, and dead ReLUs. Every number on screen is real output from code you can run.
Chapters
- 0:00 Intro
- 0:47 The building block
- 1:21 You've built one already
- 1:56 What the bias is for
- 2:25 The perceptron
- 3:06 XOR: one line isn't enough
- 3:47 Two layers solve XOR
- 4:23 Activation functions
- 5:03 Why ReLU?
- 5:43 Layers and the forward pass
- 6:32 What can go wrong: the collapse
- 7:12 Universal approximation
- 8:01 What can go wrong: dead ReLUs
- 8:44 Try it yourself
- 9:16 Interview corner
- 10:25 Recap and next stop
Cheat sheet
- Neuron = weighted sum of inputs + bias, then an activation function
- One neuron with a sigmoid = logistic regression
- The bias sets how much evidence it takes to tip over
- One layer (a perceptron) can't solve XOR; two layers can
- Sigmoid: 0 to 1, slope at most 0.25 · tanh: -1 to 1 · ReLU: max(0, z)
- Without activations, any stack of layers collapses into one linear function
- Universal approximation: enough neurons can approximate any continuous function (training isn't guaranteed to find them)
- Dead ReLU: always negative, slope 0, stops learning; fixes: sensible learning rates, leaky ReLU
Code
Try it yourself (pip install numpy)
import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) # the four possible inputs
y = np.array([0, 1, 1, 0]) # XOR: 1 when exactly one input is 1
W1 = np.array([[1, 1], [1, 1]]) # hidden layer: 2 neurons
b1 = np.array([0, -1]) # one bias per hidden neuron
W2 = np.array([1, -2]) # output neuron
b2 = 0
def relu(z):
return np.maximum(0, z) # negatives become 0
Z = X @ W1 + b1 # weighted sum plus bias
H = relu(Z) # ...then the activation
out = H @ W2 + b2 # the output neuron
for x, z, h, o in zip(X, Z, H, out):
print(f"input {x} sums {z} hidden {h} output {o}")
print("XOR solved:", np.array_equal(out, y))
# Now remove the activation: the two layers collapse into one neuron.
print("without ReLU:", Z @ W2 + b2)
W, b = W1 @ W2, b1 @ W2 + b2
print("one neuron with weights", W, "and bias", b, "gives", X @ W + b)
What we got: the hidden values come out [0 0], [1 0], [1 0], [2 1] and the outputs 0, 1, 1, 0: XOR solved. Without ReLU, the outputs become 2, 1, 1, 0, and the two layers collapse into one neuron with weights [-1 -1] and bias 2.
visual_data.pyxor_forward_pass.py
Sources
- McCulloch & Pitts, a logical calculus of nervous activity (1943)
- Rosenblatt, the perceptron (1958)
- Minsky & Papert, Perceptrons (1969), MIT Press
- Cybenko, approximation by superpositions of a sigmoid (1989)
- Hornik, Stinchcombe & White, universal approximators (1989)
- Nair & Hinton, rectified linear units (2010)
- Glorot, Bordes & Bengio, deep sparse rectifier networks (2011)
- Maas, Hannun & Ng, leaky ReLU (2013)
- Hendrycks & Gimpel, GELU (2016)
- Goodfellow, Bengio & Courville, Deep Learning, ch. 6
Transcript
Here are two rings of points, one inside the other. A neural network with ten layers learns to tell them apart, and on points it has never seen, it's right one hundred percent of the time. Now remove one tiny function from every neuron, and train it again. Suddenly it can only draw a straight line, and it scores forty-seven percent: no better than a coin flip. And depth won't save it.
A hundred layers would still draw one straight line. Which raises a classic interview question: why does a neural network need non-linear activation functions? Let's find out.
This is Just Known. From unknown, to known. Episode nine: neurons, and neural networks. Last episode wrapped up classic machine learning. Now we climb into deep learning, starting with its building block: an artificial neuron.
The idea comes from the brain, where a nerve cell fires when the signals it collects add up to enough. In nineteen forty-three, Warren McCulloch and Walter Pitts turned that into math. Today's artificial neuron does three things. Multiply each input by a weight. Add them up, plus one extra number, called the bias.
Then pass the total through an activation function. It's inspired by the brain, not a model of one. You've already built one. In episode four, our spam filter used two clues: how many digits a message has, and how long it is. As a neuron, that's two inputs, two weights, and a bias: point six three per digit, point zero zero four three per character, and minus four point five.
Take a message with ten digits and a hundred twenty characters. Weighted sum: six point eight. Add the bias: two point three. The activation here is the sigmoid, which squashes that to ninety point nine percent spam, exactly what logistic regression says. A single neuron with a sigmoid is logistic regression.
So what's the bias for? Let's delete it. Take a short message: twenty characters, no digits. With the bias, it's one point two percent spam. Without it, fifty-two.
And with no bias at all, every single message scores above fifty percent. The bias is the neuron's starting point, before any evidence. It sets how much evidence it takes to tip over: here, about seven digits. Without one, the line between yes and no is pinned to zero. In nineteen fifty-eight, Frank Rosenblatt unveiled a neuron that could learn: the perceptron.
Its activation was a step: output one if the weighted sum plus the bias is above zero, otherwise zero. Its learning rule: after each mistake, nudge every weight by the error, times its input. His team later built it as a machine, the Mark One Perceptron. Its eye was four hundred light sensors, and its weights were real knobs, turned by electric motors. Teach it AND, one only when both inputs are one, and after five passes it gets all four right.
Whenever a straight line can separate the answers, this rule is guaranteed to find one. Now try XOR, exclusive or: one when exactly one input is on. The perceptron never settles. From the second pass on, it ends every pass on the same weights, forever. We tried a million straight lines, and the best gets three of four.
The ones sit in opposite corners, and so do the zeros, so no line can split them. In nineteen sixty-nine, Marvin Minsky and Seymour Papert's book, Perceptrons, proved limits like this for single-layer perceptrons. It's often blamed for a long chill in neural network research, but about deeper networks, it offered doubts, not proofs. The fix is a second layer. Two hidden neurons both add up the inputs.
The first, with no bias, counts how many are on. The second has a bias of minus one, so it only fires when both are. Each then applies ReLU, which turns negatives into zero. Watch the four points. Zero-one and one-zero land on the same spot, and one-one moves away.
Now one straight line separates them, and the output neuron, the first count minus twice the second, gets all four right. That's the whole trick: each layer moves the data into a new space, until a straight line is enough. Let's meet the activation functions. The sigmoid, from episode four, squashes any number to between zero and one. Its cousin, tanh, squashes to between minus one and one.
And ReLU, the rectified linear unit, is the simplest: keep positives, and turn negatives into zero. What matters for learning is their slope, because training follows slopes back through every layer. The sigmoid's slope is never more than a quarter, and out at four, it's almost zero. Tanh's peaks at one. ReLU's slope is exactly one for every positive input, and zero for every negative one.
Remember that zero. That slope is why ReLU took over. On the rings, with one or two hidden layers, all three activations do fine. Stack more, and the sigmoid network falls apart: seventy-six percent with three layers, sixty with five, and fifty with ten, its loss stuck for two thousand passes. Tanh and ReLU reach one hundred percent.
Each sigmoid layer shrinks the learning signal: ten sigmoid slopes, even at their best, multiply to about a millionth. More on that in episode ten. ReLU is also cheap, and its slope never fades for positive inputs. Since papers in twenty ten and twenty eleven, it's been the standard for hidden layers. Neurons side by side make a layer, and layers in a row make a network.
Here's a real one, trained on episode eight's handwritten digits: sixty-four pixels in, a hidden layer of sixteen ReLU neurons, and ten outputs, one per digit. One thousand two hundred ten weights and biases. Running it is called the forward pass. Each layer is one matrix multiplication, plus the biases, then the activation. For this two, six of the sixteen hidden neurons stay at zero, and softmax, from episode four, turns the ten scores into probabilities: a two, with near certainty.
Training found its weights. On test images, it scores ninety-six point nine percent, and logistic regression ninety-six. On messy photos, networks pull far ahead, in episode twelve. So what can go wrong? Take the activations away, and the layers collapse.
Without an activation, a layer just multiplies by a matrix and adds a bias. Do that twice, and it's still one multiply and one add, because the two matrices multiply into one. Here are a hundred layers of random weights, seven thousand eighty-nine numbers. Multiply them out, and they become one layer: two weights and a bias. Run five points through both, and the outputs match to fifteen decimal places.
That's what happened to our ten-layer network. Without ReLU, it was secretly one neuron with a sigmoid: logistic regression. Without a non-linear activation, any network is just a linear model. Each ReLU neuron is a hinge: flat, then a ramp. Add enough hinges, and you can bend a line into almost any shape.
Here's a made-up wiggly curve, fit by one hidden layer. One neuron gives one bend, and an error of point six four. Three neurons: point two six. Ten: point zero six. Fifty: point zero two.
That's the idea behind the universal approximation theorem, proved in nineteen eighty-nine by George Cybenko, and by Kurt Hornik and colleagues. With enough neurons, one hidden layer can get as close as you like to any continuous function on a bounded range. But it only promises the weights exist, not that training will find them. From five random starts, our ten-neuron network's error ranged from point zero six to point six two. ReLU's weak spot is its flat side.
If a neuron's weighted sum is negative for every input, it outputs zero for all of them, and its slope is zero everywhere. No learning signal gets through, so it tends to stay that way: a dead ReLU. In our ring network, this neuron's sum stays below zero on all seven hundred training points. Twenty-eight of this network's eighty neurons are dead, and it still scores one hundred percent. Now raise the learning rate from point zero zero one to point one, same start.
The big steps kill sixty-three of the eighty, including three entire layers. Every point gets the same answer: fifty percent. The fixes: sensible learning rates and starting weights, or leaky ReLU, which keeps a small slope for negatives. You can run this yourself, in about twenty lines of NumPy. The code is in the description.
It sets the weights by hand, then runs the forward pass: multiply, add the biases, apply ReLU, then the output neuron. The hidden values come out zero-zero, one-zero, one-zero, and two-one, and the outputs zero, one, one, zero. XOR, solved. Then it removes ReLU. The outputs become two, one, one, zero, and the two layers collapse into one neuron: weights of minus one and minus one, and a bias of two.
Interview corner. Why does a neural network need non-linear activation functions? And why is ReLU the usual choice for hidden layers? Pause here, and try answering in thirty seconds.
Here's a strong answer. Without a non-linear activation, each layer is a matrix multiply plus a bias, and any stack of them collapses into one linear function, which can't even solve XOR. Non-linear activations let each layer bend the space, and with enough neurons, a network can approximate almost any continuous function. ReLU is the usual choice because it's cheap, and its slope is one for positive inputs, so the learning signal doesn't fade through active neurons, while the sigmoid's slope is at most a quarter. Its weakness is dead neurons, which leaky ReLU and sensible learning rates help avoid.
Sigmoid and softmax stay at the output, for probabilities. To stand out, add that one neuron with a sigmoid is exactly logistic regression, and that transformers like GPT-2 use a smooth cousin of ReLU, called GELU. Let's recap. A neuron multiplies its inputs by weights, adds a bias, and applies an activation. With a sigmoid, it's logistic regression.
One layer can't solve XOR, but two can, because each layer moves the data into a new space. Without activations, any stack collapses into one straight line. And ReLU is the default, but watch for dead neurons. We set the XOR weights by hand. A real network has thousands, or billions, and finds them itself, by working out which weight to blame for each mistake.
That's episode ten: backpropagation, how networks learn. This is Just Known. From unknown, to known.
Note: the spam messages and handwritten digits are real data; the XOR points, the two rings, the wiggly curve and the 100 random layers are made-up data. Pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.
