What is a large language model? LLMs in plain words

A large language model, or LLM, is an AI system trained on a vast amount of text to do one thing: predict what text comes next. What makes it large is the size of everything involved, billions of adjustable numbers trained on trillions of words with enormous computing power. That single skill, done at that scale, is enough to write, translate, summarize, answer questions and write code.

In short

• An LLM predicts the next piece of text; its other abilities come from doing that very well.
• "Large" means billions of parameters, trillions of training tokens, and the computing power to match.
• It is not a search engine or a database: it generates plausible text and has no built-in way to check a fact.

What does a language model actually do?

A language model gives a probability to every possible next piece of text. Shown "The cat sat on the", a good one puts a high probability on "mat" and a low one on "democracy". Text is handled as tokens, which are whole words, parts of words or punctuation, each turned into a number. To write a paragraph, the model predicts a token, adds it to the text, and predicts again.

Language models are old; phone keyboards have suggested next words for years using simple ones. What changed is what happens when the prediction gets very good. To predict the next word of a physics textbook, a legal contract or a Python program, a model has to pick up a great deal about physics, contracts and Python. Nobody programs those skills in. They emerge from training on prediction.

What makes a language model "large"?

Three things grow together.

• Parameters. These are the adjustable numbers inside the model's neural network, tuned during training. OpenAI's GPT-3, from 2020, had 175 billion. Models released since range from about a billion, small enough for a phone, to far more.
• Training data. Text from the web, books, code and other sources, measured in tokens, now often trillions of them.
• Compute. The processing power used in training, which for the largest models means thousands of specialized chips running for months.

In 2020, researchers at OpenAI found that a model's error falls in a smooth, predictable way as all three grow, a pattern known as scaling laws that held across more than seven orders of magnitude. In 2022, DeepMind's Chinchilla paper refined the recipe after training over 400 models: for a fixed budget, parameters and training data should grow in equal proportion. Its 70-billion-parameter model beat GPT-3 and a 280-billion-parameter model of its own, with the same computing budget, by being smaller and trained on four times as much data as that model.

How big is big? A worked example

Chinchilla is a useful yardstick because its numbers are public. Here is what they mean in everyday terms.

1. Training text. Chinchilla was trained on about 1.4 trillion tokens, roughly 20 tokens for each of its 70 billion parameters. In English a token is about three-quarters of a word, so that is about 1 trillion words.
2. In books. A long novel runs about 100,000 words. 1 trillion ÷ 100,000 is 10 million novels: reading one a week would take about 190,000 years.
3. In memory. Each parameter is a number stored in 2 bytes at the precision commonly used. 70 billion × 2 bytes is 140 gigabytes, several times the memory of a typical laptop, just to hold the model.
4. The output of all that. The finished model is that one file of 140 gigabytes of numbers. It is not a library of the training text to search, but patterns that let it predict text like it, though passages repeated often in the data can be memorized word for word.

That last point explains a good deal of LLM behavior. A model compresses its training into patterns, so it can usually reproduce what was common in that data and often garble what was rare.

How is an LLM different from a search engine or a database?

People often use chatbots as if they were search engines, and the difference matters.

	Large language model	Search engine	Database
What it does	Generates new text from learned patterns	Finds existing pages that match a query	Stores and returns exact records
Where answers come from	Patterns from training, up to a cutoff date	Pages crawled from the live web	Whatever was put in
Can it be wrong?	Yes, fluently, with no warning	It can surface wrong pages, but shows where they came from	Only if the data was wrong
Good for	Drafting, summarizing, explaining, rewording	Finding sources and current facts	Exact lookups such as an account balance

Many products now combine the first two. The system runs a search, pastes the results into the model's input, and has it write an answer from them, a technique called retrieval-augmented generation. That gives the model fresher and checkable material, but it is still the model doing the writing.

How does a raw model become a chatbot?

Prediction alone produces a model that continues text rather than answers it. Assistants such as ChatGPT go through further training: first on example conversations written by people, then on people's rankings of the model's answers, a process called reinforcement learning from human feedback. How ChatGPT works walks through those steps one by one. The architecture underneath nearly every modern LLM is the transformer, from a 2017 Google paper, which is what the T in GPT stands for.

What are LLMs good at, and where do they fail?

LLMs are strong at tasks where fluent, plausible text is the goal: drafting an email, summarizing a long report, explaining a concept at different levels, translating, rewording, and writing or explaining code. They also handle loosely worded requests well, which older software never could.

The weaknesses follow from the same design. A model has no built-in fact checking, so it can state false things with total confidence, which is why chatbots hallucinate. Its knowledge stops at its training cutoff unless something feeds it newer text. It can only take in so much text at once, a limit called the context window. It can repeat biases in its training data. And it can be thrown by small changes in how a question is worded.

Still argued

Researchers disagree about what LLMs are doing. One view holds that predicting text this well requires building internal models of the world, and points to studies finding such structure inside trained models. Another holds that LLMs are statistical pattern-matchers that imitate understanding, and points to simple reasoning errors no understanding reader would make. Related to this is whether scaling alone will keep making them more capable, or whether, as some prominent researchers argue, new methods will be needed.

LLMs are one part of artificial intelligence, and the best known part of generative AI. They are built from the same parts as the rest of the field: data, a neural network, and a training loop that nudges numbers until predictions improve.

Check yourself

• Without looking back: what single task is an LLM trained on, and how does that lead to skills like summarizing?
• What three things grow together to make a model "large", and what did the Chinchilla paper say about balancing them?
• Why can an LLM state a false fact fluently, when a database would not?

Sources

• Attention Is All You Need — Ashish Vaswani and others, Google, 2017
• Language Models are Few-Shot Learners — Tom Brown and others, OpenAI, 2020
• Scaling Laws for Neural Language Models — Jared Kaplan and others, OpenAI, 2020
• Training Compute-Optimal Large Language Models — Jordan Hoffmann and others, DeepMind, 2022
• Large language model — Wikipedia