Why does AI hallucinate? Why chatbots make things up

AI chatbots hallucinate, stating something false as if it were true, because a language model writes by predicting the next likely word rather than by looking facts up, and because the way models are trained and scored rewards a confident guess over an honest "I don't know". That is why AI does it most with rare facts and exact details: a made-up quote, a book that doesn't exist, a court case no judge ever decided.

In short

• A language model produces text that sounds right; it has no built-in step that checks whether the text is true.
• Rare facts, exact details such as dates and citations, and questions with no answer in its training are where it guesses most.
• Grounding answers in sources you can open reduces the problem, and checking those sources is the reader's best defense.

What does "hallucination" mean in AI?

In AI, a hallucination is fluent, confident output that is wrong or made up. The word is borrowed loosely from psychology, and some researchers prefer "confabulation": the U.S. National Institute of Standards and Technology uses that term in its 2024 guidance on generative AI risks, describing it as confidently stated but false content.

Researchers split the errors into two kinds. Some contradict the source the model was given, as when a summary says a report found a rise where the report found a fall. Others add something the source can't confirm, as when a summary includes a figure the report never gave, or an answer from memory quotes a speech that never happened. A widely cited survey of the research calls these intrinsic and extrinsic hallucinations.

The best-known case reached a federal court in New York. In 2023, lawyers in Mata v. Avianca filed papers citing six court decisions that ChatGPT had invented, complete with plausible names, docket numbers and quotes. In his opinion on sanctions of 22 June 2023, the judge imposed a $5,000 penalty on the two lawyers and their firm together and ordered them to write to each judge falsely named as the author of a fake opinion.

Why does a language model make things up?

A large language model is trained on a vast amount of text to do one thing: given the words so far, predict which word comes next. Repeated millions of times over, that simple skill produces essays, code and answers. But nothing in that process stores facts in a list it can consult. The model holds patterns of how words tend to follow each other, compressed into billions of numbers.

So the model produces the most plausible continuation, and most of the time the plausible answer is also the true one, because true statements are what most of its training text contains. The trouble starts where the two come apart. A famous fact, such as the year of the first Moon landing, appears thousands of times in training text, and the model gets it right almost every time. An obscure fact, such as the birthday of a minor academic, may appear once or never. There, the model still produces something that has the shape of an answer: a date, a title, a name.

A worked example: how a guess beats an honest blank

Imagine a model asked for a little-known person's birthday that it never saw in training. Its choices are to guess a date or to say it doesn't know. Now look at how most tests score it: 1 point for a right answer, 0 for a wrong one, and 0 for "I don't know".

Strategy	Chance of being right	Expected score on the test
Says "I don't know"	0	0
Guesses a date	1 in 365	about 0.003
Guesses, but wrong answers cost 1 point	1 in 365	about −0.995

Under the first two rows, guessing always scores higher than admitting uncertainty, so a model tuned to do well on such tests learns to guess. Across thousands of hard questions, those small gains add up on a leaderboard. Change the scoring so a wrong answer costs more than a blank, as in the last row, and the honest blank wins. This is the argument of a September 2025 paper by researchers at OpenAI and Georgia Tech, Why Language Models Hallucinate: hallucinations are partly an unavoidable result of statistics, for facts seen too rarely to learn, and partly a habit that common benchmarks reward. Its proposed fix is to change how the main tests are scored, not only to add new hallucination tests.

Why are fake citations and quotes so common?

Citations are the clearest case of plausible shape without a fact behind it. A reference has a fixed pattern: authors, a year, a title, a journal, page numbers. A model has seen millions of real ones and can produce a new one that fits the pattern perfectly. Unless the system actually searched for and read that paper, the citation is a prediction of what such a reference would look like.

The same goes for exact quotes, statistics, URLs and legal cases. They are specific, they sound authoritative, and each detail is a fresh chance to be wrong. A model can also fill a gap in a question: asked about a product feature that doesn't exist, it may describe one, because describing features is what answers to that kind of question usually do. Its tone stays the same whether it is right or wrong, which is why hallucinations are easy to miss and why careless AI output, sometimes called AI slop, spreads errors so quickly.

What reduces hallucinations?

The most effective fix is to stop asking the model to answer from memory. In retrieval-augmented generation, a system first searches a set of documents or the web, then hands the passages it found to the model and asks it to answer from them. The technique was described in a 2020 paper by researchers at Facebook AI, and chatbots that search the web before answering use the same basic idea. It narrows the problem rather than ending it: a model can still misread a source or cite one that doesn't say what it claims.

Other measures help at the edges. Models can be trained to say when they are unsure and rewarded for it. Answers can carry links to the exact passages they rely on, so a reader can check them. And a second pass can compare a draft answer against its sources and flag claims that none of them support. Rates of hallucination vary widely by model, by task and by how the test is built, so a single headline percentage tells you little without knowing what was measured.

Still argued

Whether hallucination can be eliminated is argued. One view, set out in the 2025 OpenAI paper, is that it can be greatly reduced by rewarding models for abstaining when unsure. Another holds that any system that generates text by prediction will sometimes produce false statements with confidence, so the goal is to manage the error rate, not to reach zero. Both sides agree that checking sources remains necessary.

How can you tell when an AI answer is made up?

There is no visual sign, so the check has to be deliberate. A few habits catch most problems:

• Open the source. If an answer cites a paper, a case or an article, follow the link or search for its exact title. A citation you can't find is a warning.
• Distrust precision you didn't ask for. Exact figures, dates and quotes from memory are where models slip most.
• Ask for the source text. A system that searched should be able to show the passage behind each claim.
• Ask it in a new chat. If the answer changes each time, the model is likely guessing.
• Check anything you will act on. Medical, legal and financial answers deserve a primary source, every time.

None of this means chatbots are useless. It means they are better at drafting, explaining and summarizing material in front of them than at recalling exact facts. Knowing how ChatGPT works makes the pattern clear, and artificial intelligence in general works the same way: systems learn patterns from data, and a pattern is not the same thing as a record.

Check yourself

• Without looking back: why does a model predicting the next word sometimes produce a confident falsehood?
• Why would a test that scores "I don't know" as zero push a model to guess?
• What does retrieval change about how a chatbot answers, and what problem does it leave?

Sources

• Why Language Models Hallucinate — Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, 2025
• Survey of Hallucination in Natural Language Generation — Ziwei Ji and others, ACM Computing Surveys, 2023
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis and others, 2020
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — National Institute of Standards and Technology, 2024
• Mata v. Avianca, Inc., Opinion and Order on Sanctions — U.S. District Court for the Southern District of New York, 2023