What does GPT stand for? Transformers, simply explained

GPT stands for generative pre-trained transformer. Generative means it produces new text. Pre-trained means it first learns from a huge amount of text before being shaped for any particular job. Transformer is the design of the neural network inside it, introduced by Google researchers in 2017. Together, the three words describe a model that does its writing by predicting the next word, which is what powers ChatGPT and many other chatbots.

In short

• GPT is short for generative pre-trained transformer: a model that writes, trained first on broad text, built on the transformer design.
• The transformer's key idea is attention, which lets each word in a passage look directly at the other words, all in one step.
• GPT is the model family; ChatGPT is a product built on it.

What does each word in GPT mean?

Generative. The model generates: it produces new text rather than sorting or scoring existing text. It writes by predicting one small piece of text, a token, at a time, then adding it and predicting again. That makes it a kind of generative AI, the family that also includes image and music generators.

Pre-trained. Before it is taught any specific task, the model learns from a vast amount of ordinary text by trying to predict each next word. This first stage is pre-training. Afterwards it can be fine-tuned, trained a little more on a smaller set of examples, for a particular job such as answering questions or following instructions. The point of the idea is that one expensive general training run can be reused for many tasks.

Transformer. This is the architecture, the way the model's neural network is wired. It comes from a 2017 paper by eight researchers, most of them at Google, Attention Is All You Need, written to improve machine translation.

Where did the name come from?

OpenAI published the first GPT in June 2018, in a paper titled Improving Language Understanding by Generative Pre-Training. It described exactly the two stages the name now stands for: "generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning" on each task. The model was a 12-layer transformer, and it was pre-trained on BooksCorpus, a collection of over 7,000 unpublished books.

Model	Year	Size	What it showed
GPT	2018	about 117 million parameters	Pre-training on books, then fine-tuning, beat the previous best results on 9 of 12 language tasks
GPT-2	2019	1.5 billion parameters	A larger model wrote convincing paragraphs; OpenAI released it in stages over misuse worries
GPT-3	2020	175 billion parameters	Could do new tasks from a few examples in the prompt, without fine-tuning
ChatGPT	2022	built on a GPT-3.5 model	Fine-tuning with human feedback turned the model into a conversational assistant

Parameters are the adjustable numbers a model tunes in training; large language models are measured by them. Later versions are larger and more capable, but OpenAI has published much less about their insides, so their sizes are not public facts.

What is a transformer?

A transformer is a neural network that reads a whole passage at once and lets every word weigh how much every other word matters to it. That weighing is called attention.

Before 2017, the leading language models were recurrent neural networks, which read text one word at a time, left to right, carrying a running summary forward. Google's blog post on the transformer set out the two problems with that. A decision that depends on a word far back needs many steps, and the more steps, the harder it is to learn. And reading strictly in order makes it hard to use chips that do many calculations in parallel. The transformer removed the one-at-a-time reading. Every word is processed together, and the connections between words are computed directly.

That design is what made models like GPT practical at enormous scale. Training on trillions of words is only feasible if the work can be spread across thousands of chips running side by side, and the transformer allows it.

How does attention work? An everyday example

Take the sentence: "The trophy didn't fit in the suitcase because it was too big." What does "it" mean? A person knows it's the trophy, because big things don't fit. Change one word, "because it was too small", and "it" becomes the suitcase. A model reading word by word has to have carried the right information forward. A transformer can look back directly.

Here is the idea in steps, much simplified:

1. Each word asks a question. "It" produces a query that amounts to: which earlier word am I standing for?
2. Each word offers a label. Every word in the sentence produces a key describing what it is, such as "a noun, a physical object that can be big".
3. Match and score. The query from "it" is compared with every key. "Trophy" and "suitcase" score high; "the" and "because" score low. The scores are turned into weights that add up to 1, for example 0.6 for trophy, 0.3 for suitcase and 0.1 spread over the rest.
4. Blend. "It" takes in information from each word in proportion to its weight. After this step, the model's version of "it" carries mostly the meaning of "trophy".

A real transformer does this many times over. Each layer has several attention heads, each free to learn a different kind of link, such as which noun a pronoun points to, or which verb goes with which subject, and models stack dozens of layers. In a GPT, each word can attend only to the words before it, because the model writes from left to right. Google's blog showed this happening in a translation model: one attention head picked out the two nouns "it" could refer to, and weighed them differently depending on the sentence.

The analogy that helps most is a room of people reading the same page, where each one, before deciding what their own word means, glances at everyone else's and pays most attention to the few that matter.

GPT, ChatGPT, or GPT in another sense?

Three things share the letters. GPT, the model family, is the subject of this note. ChatGPT is OpenAI's chat product, released on 30 November 2022, which runs on GPT models tuned to hold a conversation; how ChatGPT works covers the steps from model to assistant. And in computing, GPT also stands for GUID Partition Table, the standard way of dividing a hard drive into sections, which has nothing to do with AI.

Other companies' chatbots also use transformers, so "transformer" is the general term and "GPT" is OpenAI's name for its own line. For how this fits into the wider field, start from what artificial intelligence is, or go one layer down to deep learning.

Check yourself

• Without looking back: what does each of the three words in GPT mean?
• What problem with reading text one word at a time did the transformer solve?
• In the trophy sentence, how does attention help the model work out what "it" refers to?

Sources

• Attention Is All You Need — Ashish Vaswani and others, Google, 2017
• Transformer: A Novel Neural Network Architecture for Language Understanding — Jakob Uszkoreit, Google Research, 2017
• Improving Language Understanding by Generative Pre-Training — Alec Radford and others, OpenAI, 2018
• Language Models are Few-Shot Learners — Tom Brown and others, OpenAI, 2020
• Generative pre-trained transformer — Wikipedia