AI · 04 Artificial Intelligence

Yes or No? Logistic Regression & Classification

Yes or No? Logistic Regression & Classification

Why does a spam filter say "96% spam" instead of just yes? We build a real one on 5,574 real text messages. Logistic regression adds up clues into a score, the sigmoid squashes it into a probability, and a threshold makes the call. Plus: the decision boundary, log loss, softmax, SVMs in brief, why accuracy is a trap with imbalanced classes, and choosing your threshold. Every number on screen is real output from code you can run.

Chapters

Cheat sheet

Code

Try it yourself (pip install scikit-learn)
Data: unzip the SMS Spam Collection from https://archive.ics.uci.edu/dataset/228/sms+spam+collection

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

labels, texts = [], []
for line in open("SMSSpamCollection", encoding="utf-8"):
label, text = line.rstrip("\n").split("\t", 1)
labels.append(1 if label == "spam" else 0)
texts.append(text)

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2, random_state=0, stratify=labels)
vec = TfidfVectorizer() # a score for every word in a message
model = LogisticRegression() # weighted sum of those scores, squashed into a probability
model.fit(vec.fit_transform(X_train), y_train)
print(f"test accuracy: {model.score(vec.transform(X_test), y_test):.1%}")

new = ["WINNER! You have won a free prize. Call now to claim it",
"Are we still on for lunch tomorrow?",
"Your parcel is waiting. Reply to arrange delivery"]
X_new = vec.transform(new)
for text, p, y in zip(new, model.predict_proba(X_new)[:, 1], model.predict(X_new)):
print(f"{p:7.2%} spam, {'spam ' if y else 'inbox'} {text}") # predict(): threshold 0.5

words = vec.get_feature_names_out()
print("spammiest words:", ", ".join(words[i] for i in model.coef_[0].argsort()[::-1][:8]))

What we got: 96.8% accuracy; prize 96.28% spam, lunch 5.70%, parcel 49.99% (inbox, a hair under 0.5); spammiest words: txt, call, text, free, stop, mobile, www, claim. Now try class_weight="balanced": 135 of 149 spam caught instead of 114, but 11 good messages flagged instead of 1.

Sources

Transcript

A text arrives. "Winner! You have won a free prize. Call now to claim it." Before it ever reaches you, a filter has already decided: spam.

Ninety-six percent sure. Not just yes or no. A percentage. And sometimes, it's genuinely torn. This one, about a parcel, scored forty-nine point nine nine percent, and slipped into the inbox by a hair.

So how does a model turn a message into a probability? Today we build one, and it's called logistic regression. Which raises a classic interview question: why is it called regression, if it's a classifier? Let's find out.

This is Just Known. From unknown, to known. Episode four: yes or no? Logistic regression, and classification. In episode two, we split supervised learning into two kinds.

Regression predicts a number, like delivery minutes. Classification picks a category. And the simplest classification is a yes-or-no question. So let's use real data: the SMS Spam Collection, five thousand five hundred seventy-four real text messages, each labeled by people as spam, or not. About thirteen percent are spam.

We'll start with one simple clue: how many digits a message contains. Spam loves phone numbers, prices, and codes. Here's the obvious first try, from episode three: fit a straight line. Label spam as one, and not spam as zero, and draw the best line through the points. It sort of works in the middle.

But a line never stops. At thirty digits, it predicts one point four one. A hundred and forty-one percent spam? And on other data, the same line can just as easily dip below zero. A probability has to stay between zero and one, and a line can't do that.

The fix is to keep the line, but squash its output. First compute a score, exactly like before: w times the digits, plus b. Then pass that score through the sigmoid function: one over one plus e to the minus score. Whatever goes in, something between zero and one comes out. A score of zero gives exactly a half.

A big positive score gives almost one. A big negative score, almost zero. That S-shaped curve is the logistic function, and it's where logistic regression gets its name. The curve itself goes back to population models in the eighteen-thirties and forties. The word logit came in 1944, and a 1958 paper by David Cox helped make logistic regression a standard tool.

Here it is on the real messages. Logistic regression learns a score of point six five eight per digit, minus four point one four. Squash it, and a message with no digits has a one point six percent chance of being spam. Five digits, thirty percent. Ten digits, ninety-two.

The curve crosses one half at about six digits. And it matches reality. Of the messages with no digits at all, under one percent are spam. With fifteen to nineteen digits, ninety-nine point five percent are. A probability isn't a decision yet.

To get a yes or no, you pick a threshold. The usual default is one half: at or above it, call it spam. Below, let it through. That threshold is a choice, not something the model learns. Move it, and some messages change sides.

We'll come back to why you'd move it, because it matters more than it sounds. Now give the model two clues: the digits, and the message's length. The score becomes a weighted sum of both, plus b. And here's the neat part: the messages where the score is exactly zero, the fifty-fifty ones, form a straight line. That's the decision boundary.

Spam on one side, not spam on the other. With three clues, it's a flat plane. With thousands of clues, the same thing in thousands of dimensions. It's still a linear model; it just draws a straight cut through the data. To train it, we need a loss.

Mean squared error, from episode three, works badly here: through a sigmoid, it makes the landscape bumpy, and its slopes go flat exactly when the model is confidently wrong. So classification uses log loss, also called cross-entropy. Take the probability the model gave the right answer, and the loss is minus its logarithm. For a spam message, if the model said ninety percent spam, the loss is just point one. Fifty-fifty, point seven.

But one percent, confidently wrong, and the loss jumps to four point six. Log loss hates confident mistakes. And for logistic regression, its landscape is a single smooth bowl. And then, training is the same walk downhill as last time. Even the slope looks the same: for linear regression, it's the prediction minus the target, times the input.

For logistic regression, it's the probability minus the label, times the input. In practice, scikit-learn's default doesn't take plain gradient steps. It uses a method called L-BFGS, which also estimates the curvature, to take smarter steps. Same idea, fewer steps. What if there are more than two answers?

Say spam, promotion, or personal. The model gives each class its own score, and a function called softmax turns all the scores into probabilities that add up to one. Scores of two, one, and point one become about sixty-six, twenty-four, and ten percent. With just two classes, softmax boils down to the sigmoid. And you've seen it before: in episode one, softmax turned GPT-2's fifty thousand scores into the probabilities for the next token.

There's another classic way to draw that line: the support vector machine. Instead of probabilities, it looks for the boundary with the widest margin, the biggest gap between the line and the nearest points of each class. Those nearest points are called support vectors. It gives a score, not a probability, but it was a go-to classifier for years. So what can go wrong?

Only about thirteen percent of these messages are spam. That makes accuracy a trap. A lazy model that always says not spam scores eighty-six point six percent accuracy on our test set. And it catches zero spam. Our real model scores ninety-six point eight, and catches one hundred fourteen of the one hundred forty-nine spam messages.

When one class is rare, one fix is to weight it more during training. With balanced class weights, our model catches one hundred thirty-five spam messages. The price: it now flags eleven good messages instead of one. Measuring this properly is episode six. The threshold is where you choose your mistakes.

At one half, our filter catches one hundred fourteen spam messages and flags one good one. Lower it to point three: one hundred thirty caught, seven good ones flagged. At point one: one hundred forty-four caught, but ninety-one good messages lost. Which is worse, a spam in your inbox, or a job offer in your spam folder? That's a product decision, not a math one.

And trust the probabilities carefully. Large neural networks are often overconfident: they say ninety-nine percent, and are wrong more often than that. Logistic regression's probabilities are often fairly honest, but check: when it says seventy percent, is it right about seventy percent of the time? You can build this spam filter yourself, in about twenty lines of Python with scikit-learn. The code and the dataset link are in the description.

It turns each message into scores for the words it contains, then fits a logistic regression on eighty percent of the messages. On the twenty percent it never saw, it's right ninety-six point eight percent of the time. Ask it about new messages, and predict proba gives you the probabilities: the prize message, ninety-six percent spam. Lunch, under six. The parcel, forty-nine point nine nine, a hair under the threshold.

And you can ask which words push hardest toward spam: text, call, free, stop, claim. Nobody wrote those rules either. Interview corner. Why is it called logistic regression, if it's a classifier? And why don't we train it with mean squared error?

Pause here, and try answering in thirty seconds.

Here's a strong answer. Logistic regression fits a linear score, a weighted sum of the features, which models the log-odds of the positive class. That's the regression part. The sigmoid turns that score into a probability, and a threshold, often one half, turns the probability into a class. So it's a regression on the log-odds, used as a classifier.

We train it with log loss, not mean squared error. With a sigmoid, mean squared error gives a bumpy loss surface with weak gradients when the model is confidently wrong, while log loss is convex and punishes confident mistakes hard. To stand out, add that the threshold is a business choice that trades caught spam against false alarms, and that with imbalanced classes, accuracy alone is misleading. Let's recap. Logistic regression computes a linear score and squashes it with the sigmoid into a probability.

A threshold turns that into a yes or no, and the line where the score is zero is the decision boundary. It's trained with log loss, by walking downhill. And when classes are imbalanced, accuracy can look great while missing everything that matters. Our filter scored ninety-six point eight percent on messages it had never seen. But what if a model scores a hundred percent on its training data?

Believe it or not, that's usually bad news. That's episode five. This is Just Known. From unknown, to known.

Note: the messages are real (text messages, not emails); the three "new" ones we wrote. Pictures marked "illustration" are not data.
Narration voice generated with Google Gemini text-to-speech.