What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation, or RAG, is a way of making an AI language model look things up before it answers. When a question comes in, the system first searches a set of documents for the passages most likely to hold the answer, then gives those passages to the model along with the question, so the answer can draw on them and cite them. It is how many company chatbots answer from their own manuals, policies and records.

In short

• RAG is "look it up, then answer": a search step finds relevant text, and a language model writes the answer from it.
• It lets a model use information it was never trained on, including private and recent documents, without retraining it.
• It reduces made-up answers but doesn't eliminate them: the search can miss, and the model can still misread or ignore what it's given.

Why do language models need to look things up?

A large language model learns from a huge body of text, then stops learning. Everything it "knows" is stored in the numbers inside it, and three problems follow. It knows nothing after its training data ends. It knows nothing about documents it never saw, such as a company's internal policies. And when it doesn't know, it may produce a fluent, confident answer anyway, the habit explained in why AI hallucinates.

RAG answers all three by keeping knowledge outside the model. The documents stay in an ordinary searchable store, which can be updated at any time. The model's job shrinks from remembering facts to reading the right passage and writing a good answer from it, the way a person with an open book does better on a test than one going from memory.

The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University. They combined a language model with a searchable index of Wikipedia and found that the result produced "more specific, diverse and factual language" than the model alone. Lewis later said that had they known how widespread the work would become, they "would have put more thought into the name."

How does RAG work, step by step?

Most RAG systems have two phases: preparing the documents once, and answering each question.

Preparing the documents:

1. Split each document into chunks, typically a few paragraphs each, so the search can return just the relevant part.
2. Turn each chunk into an embedding: a list of numbers that captures its meaning, made so that passages about similar things get similar numbers.
3. Store the chunks and their embeddings in a vector database, a store built to find the nearest matches to a given list of numbers quickly.

Answering a question:

1. Turn the question into an embedding the same way.
2. Retrieve the few chunks whose embeddings are closest to the question's.
3. Build a prompt: the instructions, the retrieved chunks, and the question.
4. The model writes an answer from that prompt, ideally saying which chunk each claim came from.

Many systems add a keyword search alongside the embedding search, or a second step that re-ranks the results, because meaning-based search can miss exact terms such as product codes and names.

A worked example with four documents

Take a small bike shop that puts its help pages into a RAG system. The store holds four chunks:

Chunk	Text
A	Returns: unused bikes can be returned within 30 days of purchase for a full refund.
B	Returns: after 30 days and up to 60 days, unused bikes can be exchanged for store credit.
C	Repairs: tune-ups are free for one year after purchase.
D	Delivery: bikes are delivered within 5 working days in the city.

A customer asks: "I bought a bike 40 days ago and never rode it. Can I return it?"

1. The question is turned into an embedding and compared with the four chunks. The returns chunks, A and B, score closest; repairs and delivery score low. Say the system keeps the top two.
2. The prompt now reads, in effect: "Answer using only these passages. Passage A: … Passage B: … Question: I bought a bike 40 days ago…"
3. The model answers: "A full refund is only available within 30 days (A). At 40 days, an unused bike can be exchanged for store credit, which applies up to 60 days (B)."

Without retrieval, the model would have had to guess the shop's policy, and a common guess, "most stores allow 30-day returns", would have missed the store-credit option entirely. The answer is only as good as what the search hands the model. If the system had kept only one chunk and picked A, the reply would have been a flat no. If chunk B were out of date, the answer would be confidently wrong, and correct to the letter of what it was given.

How is RAG different from training the model?

RAG changes what the model reads, not what it is. The other main way to give a model new knowledge is fine-tuning: training it further on new examples, which changes its internal numbers.

	Model alone	RAG	Fine-tuning
Where knowledge lives	Inside the model, from training	In documents searched at question time	Inside the model, after extra training
Updating it	Wait for a new model	Edit the documents	Train again
Can cite where an answer came from	No	Yes, the retrieved passages	No
Good for	General knowledge, writing, reasoning	Facts that change, private documents	A style, format or narrow skill

The two are often combined: fine-tuning teaches a model how to answer, and retrieval gives it what to answer from.

Does RAG stop AI from making things up?

It reduces the problem without solving it. Retrieval gives the model real text to lean on, and a cited passage lets a reader check the claim. But failures remain: the search can return the wrong passages or miss the right one, the documents themselves can be wrong or out of date, and the model can still misread a passage, blend two together, or add details that aren't there.

A 2024 study from Stanford researchers tested this on legal research tools sold with claims of being "hallucination-free". The tools, built on RAG, made fewer errors than a general-purpose chatbot, but still gave false or misleading answers between 17% and 33% of the time on the researchers' set of legal questions. The lesson the authors drew is that RAG lowers the error rate but doesn't remove the need to check.

RAG is now a building block in larger systems. An AI agent may decide for itself when to search and what for, and standards such as the Model Context Protocol let an assistant reach live sources, such as a company's files, through the same kind of retrieval. Both are ways for artificial intelligence to work from evidence rather than memory alone.

Check yourself

• Without looking back, describe the steps a RAG system takes between receiving a question and answering it.
• Why can a RAG answer be wrong even when the model reads its passages correctly?
• When would fine-tuning be a better choice than RAG, and when would RAG be better?

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020
• Retrieval-Augmented Generation for Large Language Models: A Survey — Gao et al., 2023
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools — Magesh et al., Stanford University, 2024
• What is retrieval-augmented generation aka RAG — NVIDIA, 2025