Why AI makes things up and how to reduce it
Seven reasons why language models invent answers, seven kinds of fabrication with ways to catch them, eight ways to make it rare and an instruction with a check by code.
In short
A language model makes things up because it does not look facts up — it continues text with the most plausible words. When the answer is not in what it was given, it still produces something that sounds right: an invented clause of a contract, a price that never existed, a link to a page that is not there. It cannot be switched off completely, but it can be made rare and harmless: give the model your sources, allow it to say “I don’t know”, require a quote for every statement, check the quotes with code and leave numbers to calculations. For important decisions — the last word belongs to a person.
Why models make things up: 7 reasons
-
It predicts, not checks
The model chooses plausible words; whether the statement is true, it does not know.
-
The answer is not in the context
Without your documents the model fills the gap from general memory — or from imagination.
-
It was told to always answer
An instruction without permission to say “I don’t know” pushes it to invent.
-
Knowledge is outdated
The model learned on data up to a certain date and knows nothing about your new prices.
-
A leading question
“Why is your delivery free?” — the model is likely to agree even if it is not free.
-
Numbers, names and links
Exact details are the weakest spot: they look plausible in any form.
-
Too much in one request
In a very long context the model loses details from the middle and mixes facts.
What fabrication looks like and how to catch it
Seven kinds of errors that come up in business products, with a way to catch each one.
| Kind | Example | How to catch |
|---|---|---|
| An invented fact | “Returns are accepted within 30 days” instead of 14 | a quote from the source |
| An invented number | a price that is not in the price list | numbers from the database, not from the model |
| An invented link | a page of the site that does not exist | links only from a list |
| An invented feature | “Yes, our app can do that” | a description of what the product can do |
| Mixed objects | specifications of one model given for another | one product per piece of context |
| Outdated information | last year’s terms of delivery | a date next to every source |
| Confidence without grounds | a firm answer to an unclear question | permission to ask back |
8 ways to make fabrication rare
-
01
Give the sources
The model answers from found pieces of your documents, not from memory.
-
02
Allow “I don’t know”
A separate field for “not found” and a clear next step — a manager.
-
03
A quote for every statement
The model copies the sentence the answer is based on.
-
04
Check the quotes with code
A quote that is not in the source means the answer is not shown.
-
05
Numbers from the database
Prices, stock and dates are inserted by code, the model only words the text.
-
06
A strict format
An answer in JSON by a schema is easier to check than free text.
-
07
Small tasks
One question, a few relevant pieces — fewer chances to mix things up.
-
08
A person for important things
Contracts, money and health — the model prepares, a person decides.
An instruction and a check: 2 examples
The instruction allows the model not to know and requires quotes; the code does not trust the model and checks each quote.
The instruction to the model
“Not found” is a separate field, not a special phrase: it cannot be confused with an answer.
# The beginning of every request — the same each time, so it goes to the cache
You answer customers of the shop using only the sources below.
Rules:
1. Use only facts from the sources. Add nothing from yourself.
2. Copy the sentence that supports the answer into quotes, word for word.
3. If the sources do not contain the answer, set found to false
and say that the question goes to a manager.
4. Prices, dates and numbers — only as written in the sources.
Answer in JSON:
{"found": true, "answer": "...", "quotes": [{"source": 1, "text": "..."}]}Checking the quotes
Ten lines of code catch most invented answers before the customer sees them.
// Every quote must literally occur in the source it refers to —
// otherwise the answer is not shown and the question goes to a person
type Answer = {
found: boolean;
answer: string;
quotes: { source: number; text: string }[];
};
export function verified(a: Answer, sources: string[]): boolean {
if (!a.found) return true; // an honest “not found” needs no check
if (a.quotes.length === 0) return false;
return a.quotes.every((q) => sources[q.source - 1]?.includes(q.text.trim()) ?? false);
}What does not help against fabrication
-
“Do not make things up” in the instruction
Without sources and permission not to know, the request alone changes little.
-
Only the most expensive model
Strong models invent less often, but just as convincingly.
-
Lowering the temperature only
Answers become more uniform, not more true.
-
Asking the model to check itself
It helps a little; a check by code against the source helps much more.
-
Trusting the confident tone
An invented answer sounds exactly as confident as a true one.
-
Testing on easy questions
Fabrication appears on questions without an answer — they must be in the test set.
Questions about AI fabrication
What is an AI hallucination?
A confident answer that is not supported by facts — invented by the model in place of missing knowledge.
Can hallucinations be removed completely?
No, but sources, quotes and checks by code make them rare, and the rest are caught before the customer.
Which model invents least?
It changes from version to version; the way the product is built matters more than the model.
Does RAG solve the problem?
It removes the main cause — no data; together with a threshold and quotes it gives reliable answers.
Why does a model invent links?
A link is just plausible text for it; links are taken only from a list of real pages.
How to measure how often it invents?
A test set with questions that have no answer in the base: the share of honest “not found” is the measure.
Online form
An assistant
that does not invent
I build assistants that answer only from your sources, with a quote for every statement — and hand the question to a person when the answer is not there. Tell me about the task — I answer within one working day.