How does ChatGPT work? From prediction to conversation

ChatGPT does one thing at its core: it predicts text. Underneath it is a large language model, a program trained on a huge amount of writing to guess which small piece of text, called a token, comes next. To answer you, it predicts one token, adds it to the conversation, and predicts again, hundreds of times. Extra training with human feedback is what makes that next-word guesser work as an assistant that follows instructions.

In short

• The core skill is next-token prediction, learned from a vast amount of text, before any chatting is taught.
• Fine-tuning on example conversations and on people's rankings of answers shapes the predictor into a helpful assistant.
• It does not look answers up by default or remember you between chats: each reply is generated fresh from the conversation in front of it.

What is ChatGPT, exactly?

ChatGPT is a product made by OpenAI, released on 30 November 2022. It is a chat window on top of a large language model from OpenAI's GPT family; the first version used a model from the GPT-3.5 series. The name comes from that family: GPT stands for generative pre-trained transformer. The product has changed many times since launch, with new models, voice, image input and web search, but the way it produces a reply has stayed the same.

Step 1: Text becomes tokens

A model doesn't read letters or whole words. It reads tokens: common words, pieces of longer words, punctuation and spaces, each mapped to a number. "Unbelievable" might be split into "un", "believ" and "able". In English, a token averages about three-quarters of a word. Everything the model handles, your question and its answer alike, is a sequence of these numbers, and everything it is limited by, such as how long a conversation it can take in, is counted in them.

Step 2: Pretraining teaches it to predict

The model is a neural network with billions of adjustable numbers, called parameters. In pretraining, it is shown a vast amount of text from the web, books and other sources, one stretch at a time, and asked to predict each next token. When it guesses wrong, its parameters are nudged so the right token becomes a little more likely. Repeated over trillions of tokens, this simple task forces the model to absorb grammar, facts, styles of argument and patterns of reasoning, because all of them help predict what comes next.

OpenAI's 2020 paper on GPT-3 showed how far this goes at scale: a model with 175 billion parameters could translate, answer questions and do simple arithmetic from a few examples in the prompt, without being trained for any of those tasks. The architecture that made such scale practical is the transformer, from Google's 2017 paper Attention Is All You Need, which lets each token weigh all the tokens before it in its context at once.

A pretrained model is not yet an assistant. Ask it "What is the capital of France?" and it might continue with "What is the capital of Germany?", because lists of quiz questions are common text.

Step 3: Fine-tuning and human feedback make it an assistant

OpenAI described the method in its 2022 InstructGPT paper, and used the same approach for ChatGPT. It has three parts:

1. Show it good answers. People write example conversations, playing both the user and the assistant, and the model is trained on them. This is supervised fine-tuning.
2. Teach a judge. The model writes several answers to the same prompt, and people rank them from best to worst. A second model, the reward model, is trained to predict those rankings.
3. Practice against the judge. The assistant generates answers, the reward model scores them, and the assistant is adjusted toward higher scores. This is reinforcement learning from human feedback, or RLHF.

The effect is large. In the InstructGPT paper, people preferred answers from a tuned model with 1.3 billion parameters to those of the untuned 175-billion-parameter GPT-3, a model more than 100 times its size. Human feedback doesn't add much knowledge; it changes what the model does with the knowledge it has: answer the question, follow the instruction, decline some requests.

Step 4: Writing a reply, one token at a time

When you send a message, ChatGPT doesn't send the model only your last line. It sends the whole conversation so far, plus hidden instructions from OpenAI, called a system prompt, as one long sequence of tokens. The model reads all of it and outputs a probability for every token it knows. Here is an illustrative example, with made-up but realistic numbers, for the text "My favorite breakfast is":

Next token	Probability
" pancakes"	0.20
" eggs"	0.18
" oatmeal"	0.09
" toast"	0.08
every other token	0.45

The software then picks one token. If it always took the top choice, the writing would be flat and often repetitive. So it samples: "pancakes" is picked about 20% of the time, "eggs" about 18%, and so on. A setting called temperature controls how adventurous the sampling is. Low temperature sticks close to the top choices; high temperature spreads the picks out. Stephen Wolfram, in his walk-through of ChatGPT, notes that a temperature around 0.8 tended to read best for essays, and that this is a finding from practice rather than theory.

The chosen token is added to the sequence, and the whole process runs again for the next one, until the model produces a token that means "stop". That is why answers appear word by word, and why the same question can get a different answer each time.

What ChatGPT doesn't do

The context window is the amount of text, in tokens, the model can take in at once. Anything outside it, such as an early part of a very long chat, the model cannot see. It also has no memory between conversations of its own. When ChatGPT seems to remember you, the product has saved notes and slipped them back into the context.

By default, the model looks nothing up either. It answers from patterns learned in training, which stop at a cutoff date, and it produces the most plausible continuation, not a checked one. That is the root of AI hallucinations: confident, fluent answers that are wrong. When ChatGPT runs a web search, the search results are pasted into the context and the model writes from them, a technique called retrieval-augmented generation. It is more grounded, but it can still misread a source.

Still argued

Whether a system trained only to predict text understands anything is disputed. Some researchers argue that predicting text this well requires an internal model of the world the text describes. Others argue it is sophisticated pattern-matching over word statistics, and point to errors no understanding reader would make. The output is the same either way, so the debate turns on what "understanding" should mean, and no experiment has settled it.

Why the steps matter for using it

Knowing the pipeline explains the advice people give about chatbots. Detailed prompts work better because the whole prompt shapes every prediction. Facts, figures and quotes need checking because nothing in the process checks them. Rephrasing or regenerating gets a different answer because each reply is sampled. For the wider picture of where this fits, start with what artificial intelligence is.

Check yourself

• Without looking back: what does a pretrained model learn to do, and why isn't that enough to make an assistant?
• What are the three parts of training with human feedback, and what does the reward model learn?
• Why can the same question get two different answers a minute apart?

Sources

• Training language models to follow instructions with human feedback — Long Ouyang and others, OpenAI, 2022
• Language Models are Few-Shot Learners — Tom Brown and others, OpenAI, 2020
• Attention Is All You Need — Ashish Vaswani and others, Google, 2017
• What Is ChatGPT Doing … and Why Does It Work? — Stephen Wolfram, 2023
• ChatGPT — Wikipedia