AI · 01 Artificial Intelligence

What Happens When You Ask ChatGPT a Question?

What Happens When You Ask ChatGPT a Question?

What really happens in the few seconds after you ask ChatGPT a question? Your text is split into tokens, each token becomes a list of numbers, those numbers flow through a stack of transformer layers, and out comes a probability for every possible next token. One gets picked with a weighted roll of the dice, and the loop starts again, hundreds of times per answer. We follow the whole pipeline with real numbers from GPT-2, then look at where those probabilities come from (training), why models hallucinate, and how to answer the interview question.

Chapters

Cheat sheet

Code

Try it yourself (Python, runs on a laptop CPU)
pip install torch transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt").input_ids
print("tokens:", [tok.decode(i) for i in ids[0]])

with torch.no_grad():
logits = model(ids).logits[0, -1] # one score for every token in the vocabulary
probs = torch.softmax(logits, dim=-1) # scores to probabilities that add up to 1

top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
print(f"{tok.decode(i)!r:12} {p.item():.1%}")

for seed in (1, 2):
torch.manual_seed(seed)
out = model.generate(ids, max_new_tokens=12, do_sample=True, temperature=0.8, pad_token_id=tok.eos_token_id)
print(f"seed {seed}:", repr(tok.decode(out[0][ids.shape[1]:])))

What we got: ' the' 8.5%, ' now' 4.8%, ' a' 4.6%, ' France' 3.2%, ' Paris' 3.2%. Can you find a prompt where GPT-2 puts Paris first?

Sources

Transcript

You type a question into ChatGPT. Why is the sky blue? You press Enter, and an answer starts flowing in, a few letters at a time. Behind that window, there's no database of ready-made answers. There's a program doing one thing, over and over: guessing the next small piece of text.

In the next few seconds, your question will be chopped into pieces, turned into long lists of numbers, and pushed through dozens of layers of math. Then the answer comes out one piece at a time, each one picked with a weighted roll of the dice. It's also a favorite interview question: how does an LLM actually generate text? Let's take the whole trip.

This is Just Known. From unknown, to known. Episode one of our new series on artificial intelligence: what happens when you ask ChatGPT a question? First, the words, because people use them loosely. Artificial intelligence is the big circle: machines doing things that seem to need intelligence.

The term dates back to 1955. Inside it is machine learning: instead of writing the rules by hand, you show the machine examples, and it learns the rules itself. Inside that, deep learning: machine learning with neural networks, many layers deep. And inside that, generative AI: models that create new text, images, or audio. ChatGPT lives in that inner circle.

Strictly speaking, it's an app. The engine inside is a large language model, an LLM, and OpenAI swaps in newer engines over time. Today is about the engine. Here's the surprising part. Under all that, an LLM does just one thing.

You've seen a simple version on your phone: start typing, and the keyboard suggests the next word. An LLM is that idea, scaled up enormously. Give it some text, and it gives back a probability for every piece of text that could come next. Pick one, add it to the end, and ask again. Every answer, every poem, every line of code is built that way.

So here's today's map. Text becomes tokens. Tokens become lists of numbers. Those flow through a stack of transformer layers. Out come probabilities.

One gets picked. And the loop starts again. But if it only continues text, how does it hold a conversation? Before your question reaches the model, the app wraps it in a transcript. First, hidden instructions from the app, often called the system prompt.

Then your message, marked as the user's. Then a marker that says: assistant, your turn. Special tokens you never see separate each part. Anything else the app adds, like saved memories or web results, goes in the same transcript. So the model isn't really chatting.

It's continuing a document that looks like a chat. And when it's said enough, it writes one more special token that means: I'm done. Stop one: tokens. The model doesn't read letters. It reads tokens: common words, pieces of words, punctuation.

Why is the sky blue becomes six tokens: why, is, the, sky, blue, and the question mark. In GPT-2, a word's leading space is part of its token. Each token is just an ID number from a fixed list called the vocabulary. In GPT-2's, sky is token 6766. Rarer words get split.

Even the word tokenization becomes two pieces: token, and ization. GPT-2 knows about fifty thousand tokens. Newer OpenAI models use around two hundred thousand. A token averages about three quarters of an English word, and tokens are what AI companies count, and charge for. Tokenizers get their own deep dive in episode 14.

Stop two: embeddings. An ID number by itself means nothing. Token 6766 isn't bigger or better than token 30. So each token is swapped for a long list of numbers, called an embedding. In GPT-2, that's 768 numbers per token.

In the biggest GPT-3, more than twelve thousand. Think of each list as a point in space, with hundreds of directions instead of three. During training, tokens used in similar ways drift close together: sky near cloud, Paris near France. Meaning becomes geometry. And since order matters, the model also mixes in where each token sits in the sentence.

Embeddings get the rest of episode 14. Stop three: the transformer, the T in GPT. This is where most of the computation happens. The token vectors climb through a stack of layers. The small GPT-2, the one we'll run later, has twelve.

The biggest GPT-3 has ninety-six. Each layer does two things. First, attention. Every token looks back over itself and the tokens before it, and weighs which ones matter right now. Blue looks back and finds sky.

A word like it can work out what it refers to. Second, every vector passes, separately, through the same small neural network, which refines it. Layer by layer, blue stops being just a word, and becomes blue, the color of the sky, in a question about why. At the top, the last token's vector holds what the model needs to guess what comes next. Attention gets its own episode, episode 16.

So where does the model's knowledge live? Not in a database of facts. It lives in its parameters: numbers inside every layer that decide how strongly each signal passes through. Think of them as knobs. The small GPT-2, from 2019, has about 124 million.

The biggest GPT-3 had 175 billion. The largest Llama 3.1 model, from Meta, has 405 billion, and its weights are free to download, under Meta's license. The models behind ChatGPT? OpenAI doesn't publish their sizes. One knob means almost nothing.

But set billions of them just right, and the machine turns text into good guesses. Setting them is called training. How a network of knobs can learn at all starts in episode 9. Stop four: probabilities. That final vector becomes one score for every token in the vocabulary: 50,257 scores for GPT-2.

A function called softmax turns them into probabilities that add up to one hundred percent. Here are real numbers from GPT-2. Give it: the sky looks blue because. Its top guess is of, at 28 percent. Then the, at 17.

Then it, at about 11. Then a long tail of tiny chances. Notice what the model hasn't done. It hasn't chosen anything. It has only said how likely each next token is.

Stop five: picking one. The simplest rule is to always take the top token. That's greedy decoding. It's predictable, but on long answers it can sound flat, and get stuck repeating itself. So most chatbots roll weighted dice instead.

Of gets a 28 percent chance, the gets 17, and so on. Usually a likely token wins. Sometimes, a less likely one does. That's why the same question can get two different answers. Many models let developers tune that roll with a setting called temperature.

Lower makes the favorites stronger and the output steadier. Higher evens out the odds: more variety, and eventually, nonsense. Sampling gets a closer look in episode 18. Now the loop. The chosen token is added to the end, and the whole model runs again for the next one.

And the next. A five-hundred-word answer means hundreds of passes through the entire model. There's one shortcut. The first pass reads your whole prompt at once. After that, each pass only processes the newest token, because the work on earlier tokens is saved in a memory called the KV cache.

That's why answers stream in. You're watching the loop run, until the model writes its I'm done token. Many newer models first write reasoning tokens, a scratchpad that's usually hidden, or shown only as a summary. Same loop, just longer. But where do those probabilities come from?

From training, in two big stages. First, pretraining. The model reads an enormous amount of text: web pages, books, code. Over and over, it guesses the next token, checks the real one, and every knob gets nudged slightly toward the right answer. Across trillions of tokens, those nudges add up to grammar, facts, style, and some reasoning.

But that gives you a document-completer, not an assistant. Ask it a question, and it might reply with three more questions. So second, it's fine-tuned on example conversations, then on feedback about which answers are better. And one distinction interviewers love: training versus inference. Training sets the knobs, over weeks or months, on huge clusters of computers.

Inference uses the frozen knobs to answer you, usually in seconds. When you ask ChatGPT something, the knobs don't change. The full training story is in episode 17. So what can go wrong? Look at what the model is trained to produce: likely text.

Not true text. Usually those are the same. When they aren't, you get a hallucination: an answer that's fluent, confident, and wrong. In 2023, lawyers in New York filed a court document citing cases that didn't exist. ChatGPT had invented them, complete with quotes and citations.

A federal judge fined the lawyers and their firm five thousand dollars. Why does it happen? A 2025 paper from OpenAI argues that training and testing reward guessing. On a multiple-choice exam, a guess might score a point, but I don't know always scores zero. So models learn to bluff.

The fixes come later in the series: letting the model look things up in trusted sources, which we'll build in episode 20, making it cite them, and checking anything that matters. Fluent is not the same as true. You can run this yourself, on a laptop, for free. GPT-2 is small, and its weights are public. With Python and Hugging Face's transformers library, it takes about twenty lines.

The code is in the description. Give it: the capital of France is. The top guess is the, at eight and a half percent. Then now, then a. Paris shows up fifth, at about three percent.

To be fair, the could still lead to the city of Paris. But GPT-2 is a small model from 2019, and it's far from sure. Now let it write, twice, with two random seeds. First: the capital of France is now a haven for the unemployed. Second: the capital of France is not part of Greece or of Spain.

Both are fluent and grammatical. The first is made up. The second is true, but useless. And neither one says Paris. Interview corner.

How does a large language model like ChatGPT generate a response? Pause here, and try answering in thirty seconds.

Here's a strong answer. The text, including the system prompt and the conversation so far, is split into tokens. Each token becomes an embedding vector, and the model also tracks its position. A stack of transformer layers uses attention so every token can gather context from the ones before it. The model then outputs a probability for every token in its vocabulary.

One is chosen, either the top one or a sampled one, shaped by the temperature. It's appended, and the loop repeats until a stop token. The probabilities come from billions of parameters, learned in pretraining on next-token prediction, then fine-tuned into an assistant. To stand out, add why it can hallucinate: it's trained to produce likely text, not verified text. Let's recap.

An LLM does one thing: predict the next token, pick one, repeat. Text becomes tokens, then embeddings, then flows through transformer layers that use attention to gather context, and out come probabilities for the next token. And everything it learned lives in billions of trained knobs, which is why it can sound right without being right. Today we flew over the summit. Now we'll climb to it from the bottom: machines that learn simple patterns from data, then neural networks, then transformers, then the agents built on top.

By the end, you'll be able to explain every stop on today's map. Next stop, the base of the climb. What is machine learning, really, and how can a machine learn a rule that nobody wrote? That's episode two. This is Just Known.

From unknown, to known.

Note: every GPT-2 number on screen is real output from GPT-2 small (124M) run with Hugging Face transformers; pictures marked "illustration" are not data. ChatGPT is a trademark of OpenAI; this channel is independent and not affiliated with OpenAI.
Narration voice generated with ElevenLabs.