Topic 1.9
Hallucination from First Principles
In one line
A model predicts likely text, not true facts, so it can produce fluent, confident answers that are false. Knowing the causes tells you the defences: ground answers in sources, ask for evidence, allow I do not know, and verify what matters.
Think of it like this
A student who never says "I don't know". Asked about a book they have not read, they do not stay silent. They produce a smooth answer that sounds like what such a book would say. The answer is fluent, confident and partly invented. A model's training rewards exactly this kind of fluent continuation.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Hallucination
- A fluent, confident reply that is false or unsupported.
- Grounding
- Giving the model source text and asking it to answer from that text.
- Citation / quote
- The exact words from the source that support a claim.
- Verification
- Checking a claim with something other than the model: code, a database or a person.
- Sycophancy
- Agreeing with the user's claim even when it is wrong (see Topic 1.7).
- Fallback answer
- A fixed reply such as I could not find that, used when evidence is missing.
Step by step
01Step 1: see the failure shape
Ask about a fact that is not in the document: "Does the Pune office have a gym?" A model under pressure may answer "Yes, on the second floor". The sentence is smooth and wrong.
02Step 2: ground and ask for a quote
The prompt gives the document and requires two things for each answer: the answer and the exact words from the document that support it. In the example, SOURCE is the document and the scripted replies stand in for the model.
03Step 3: check the quote with code
quote in SOURCE is a plain string test. If the model's quote is not in the document word for word, the answer is rejected. The gym quote fails, so the invented answer never reaches the user.
04Step 4: allow an honest don't know
The third question has no answer in the document. The model replied NOT_IN_SOURCE, and the program turns that into a polite message. This is a success: the system avoided a hallucination. If your prompt never permits it, the model will invent.
05Step 5: know what the check misses
The fourth question had the right answer (6 pm) with a sloppy quote (weekends instead of weekdays), so it is rejected. Strict checks cause false rejections, which you tune by retrying or showing the source. The opposite failure also exists: a real quote used for a wrong conclusion passes the check. Phase 4 and Phase 9 add stronger validation and measurement.
Try it yourself
- 1
Fix the sloppy quote
Change the fourth quote to
closes at 6 pm on weekdays. Does the answer pass now? What does it tell you about how exact the model's quotes need to be? - 2
Make the invention pass
Add the sentence
There is a gym on the second floor.toSOURCE. The earlier invention now passes the check, because it is true to the source. Grounding checks truth against the document, not against the world. - 3
Count the outcomes
Add two more scripted rows of your own: one grounded, one that should be rejected. Predict the final counts line before running.
Code & diagrams
The model is a scripted fake with four replies. The check is real code that you can use with a real model.
Expected output
Q: When does the Pune office open?
[grounded] 9 am
Q: Does the Pune office have a gym?
[rejected] I could not verify that, so I will not answer.
Q: Who manages the Pune office?
[honest] I could not find that in the document.
Q: When does the Pune office close?
[rejected] I could not verify that, so I will not answer.
grounded=1 honest=1 rejected=2Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust any quote
In the Python verify, change the quote test to if quote != "": (accept any non-empty quote without searching the source).
Myth vs fact
Myth
If the model sounds sure, it is right.
Fact
Confidence in wording says nothing about accuracy. Hallucinations are delivered in the same tone as correct answers.
Myth
Better prompts or bigger models remove hallucination.
Fact
They reduce it. Grounding, checks and the freedom to say I do not know are what make it manageable.
Myth
If I give the model the document, it cannot hallucinate.
Fact
It can still misread, mix passages or add details. Ask for quotes and check them.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure it: build a set of questions with known answers, including questions whose answer is not in the source, and track the share of wrong, unsupported and correctly refused answers per release (Phase 9). Without the not-in-source cases you only measure half the problem.
- ▸
Quote checking can be hardened: normalise whitespace and case, require quotes of at least a few words, and for numbers check that the number in the answer appears in the quote.
- ▸
Some failures come from tools, not the model: a retrieval step that returns the wrong passage makes a faithful model answer wrongly. Log which passages were given for each answer so you can tell the cases apart (Phase 11).
- ▸
Citations that look right but point to documents that do not support the claim are common. Verify that the cited passage entails the claim, not just that it exists.
Remember this
- 1
Why it happens. Remember how a model works (Topic 0.2): it picks a likely next token, over and over. "Likely" means it looks like text the model has seen, not that it was checked. A made-up quote, a book that does not exist, or a wrong date can be perfectly likely text. There is no internal fact-checker, and no built-in sense of "I am unsure" that always shows up in the words.
- 2
Common causes. (1) Missing knowledge: the fact is rare, private, or after the cutoff (Topic 1.7), and the model fills the gap with a plausible guess. (2) Pressure to answer: training rewards helpful, complete replies, so "I don't know" is under-produced. (3) Long or noisy context: the right fact is buried (Topic 1.5) or contradicted by other text. (4) Leading questions: "Why is X true?" invites an explanation of a false X. (5) Exact detail: numbers, names, citations and URLs are easy to blur.
- 3
Fluent is not true. A hallucination does not look different from a good answer: same tone, same confidence. You cannot spot it by style. Users trust fluent text, so the failure is quiet (Topic 0.4): nothing crashes, and a wrong answer is delivered politely.
- 4
Defence 1: grounding. Give the model the facts and tell it to answer only from them (Phase 8). Retrieval brings the right text into the prompt, so the model copies rather than recalls. It lowers hallucination a lot and does not remove it, since the model can still misread or add to the text.
- 5
Defence 2: evidence you can check. Ask for a short quote from the source for each claim, and check by code that the quote really appears in the source (the example does exactly this). A made-up quote fails the check. It also catches correct answers with sloppy quotes, which you handle by retrying or by showing the quote to the user.
- 6
Defence 3: permission to say I do not know, and verification. Tell the model that when the sources do not contain the answer it must say so in a fixed form (the example uses
NOT_IN_SOURCE) and treat that as a success, not a failure. For anything with consequences, add a verification step: code computes numbers, a database confirms identifiers, a human reviews. Measure hallucination rates with an evaluation set (Phase 9) instead of trusting impressions.
Explain it without notes
Why do language models hallucinate?
Name four situations that raise the chance of hallucination.
How does asking for a quote and checking it with code reduce hallucination?
Why must your prompt allow the model to say it does not know?
Practice
Write quote_ok(quote, source) that returns True when quote is non-empty and appears in source ignoring case and extra spaces. Test it with "Opens at 9 AM" against "The office opens at 9 am.", and an empty quote.
Write summarise_outcomes(statuses) that counts how many times each status appears, returning a dictionary in first-seen order. Use ["grounded", "rejected", "grounded", "honest"].
Trade-offs
- ↔
Strict quote checks catch invention and reject some correct answers with sloppy quotes.
- ↔
Grounding lowers hallucination and needs a retrieval system and more input tokens.
- ↔
Allowing I do not know improves honesty and can annoy users if the model abstains too often, so measure both.
Done when you can
Done when you can explain why fluent text is not evidence of truth.
Done when you can list four causes of hallucination.
Done when you can write a quote-verification guard.
Done when your prompts include a safe way to say the answer is not in the sources.