Topic 8.6
Citing Sources and Refusing Without Evidence
In one line
A grounded answer names the numbered source it came from, your code checks that the citation is real and supports the claim, and when the documents do not contain the answer the system says so instead of guessing.
Think of it like this
Footnotes in a school essay. A claim with a footnote can be checked: the teacher opens the book at that page. A claim without one is just the student's word. And the honest answer to a question you never studied is "I don't know", not a confident guess. A grounded system needs both habits: cite, and admit when there is nothing to cite.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Citation
- A marker in the answer, such as [2], that points to the source a claim came from.
- Grounded answer
- An answer based on the supplied sources, with claims that can be checked against them.
- Refusal
- A fixed reply saying the documents do not contain the answer.
- Hallucination
- A fluent statement the model produced that is not supported by its input or by reality.
- Threshold
- A cut-off value: scores below it lead to refusal.
- Support check
- Testing whether a claim is actually backed by the cited source.
Step by step
01Step 1: number the sources
The prompt lists chunks as [1], [2] and the system message says: answer only from the sources, cite like [1], and if the answer is missing reply with exactly one fixed sentence.
Answer using only the numbered sources. Cite the source like [1] after each claim.
If the sources do not contain the answer, reply exactly: Not in the documents.
Sources:
[1] Unused bikes can be returned within 30 days with the receipt for a full refund.
[2] Standard shipping takes 3 to 5 days.
Question: How long do I have to return a bike?02Step 2: parse the citations
Scan the answer for [ digits ]. A hand-written scan avoids regex surprises and is easy to test. Collect the numbers in order.
03Step 3: apply cheap checks
No citation, a refusal that is not the fixed sentence, a citation outside 1..n, and numbers in the answer that are missing from the cited source. The first example runs six replies, from careful to careless, through these checks.
04Step 4: refuse early when retrieval is empty
The second example puts a threshold on the best retrieval score and prints the decision at two thresholds. Both mistakes appear: answering a question the documents do not cover, and refusing one they do.
05Step 5: see it with a real model
The two labs below send a real model the numbered sources. In the first the answer is in source [1]; in the second the answer is in neither source. Check the goals: cites [1], and refuses.
Try it yourself
- 1
Catch a new lie
Add a seventh reply, "Shipping is free over 800 dollars [2].", and predict the verdict before you run. Which check catches it, and why does 800 not appear in source 2?
- 2
Move the threshold
In the second example, try thresholds 0.50 and 0.72. Which questions change? "how much is a lock" and "how much does it cost" get exactly the same score, so can any threshold treat them differently? (It cannot, and that is the lesson.)
- 3
Use both labs
In the first lab, ask a question about both topics ("How long do returns and express shipping take?"). Does the reply cite [1] and [2] correctly? Run your Python checker idea on it by hand.
Code & diagrams
Six scripted replies stand in for a model. The checker needs no model to run.
Expected output
reply 1: ok Unused bikes can be returned within 30 days [1].
reply 2: BAD (number 60 is not in the cited source) Bikes can be returned within 60 days [1].
reply 3: BAD (cites [3] but there are only 2 sources) Express shipping costs 12 dollars [3].
reply 4: BAD (no citation) Returns are free for everyone.
reply 5: ok (refusal) Not in the documents.
reply 6: BAD (number 1 is not in the cited source; number 2 is not in the cited source) Express shipping takes 1 to 2 days [1].Uses the fake embedder from Topic 8.3. Scores are rounded to four decimals.
Expected output
question best document at 0.30 at 0.80
get my money back 1.0000 returns answer answer
how long is the battery warranty 0.8575 warranty answer answer
how much does it cost 0.7071 shipping answer refuse
how much is a lock 0.7071 shipping answer refuse
do you sell locks 0.0000 returns refuse refuse
what is the weather today 0.0000 returns refuse refuseTry: Send it as it is. Read the reply: does it answer 30 days and cite [1]? Then change the question to one about delivery and check that it now cites [2]. Try removing the instruction to cite and see what changes.
Goals
- It cites source [1]
- It says 30 days
- It does not cite the unrelated source [2]
Sign in to send this request to a real model (free).
Try: The sources say nothing about locks. Send it and check that the model replies with the fixed refusal. Then weaken the system message (delete the refusal rule) and send again: does it start to guess?
Goals
- It replies Not in the documents
- It is short (12 words or fewer)
- It does not invent a citation
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust any citation
In the first example, make the support check do nothing: in Python replace if t.isdigit() and t not in evidence: with if False:. Run it.
Myth vs fact
Myth
If the model cites a source, the answer is correct.
Fact
A citation is a claim, not proof. Models attach citations to wrong statements. Check that the number exists and that the source supports the claim.
Myth
Telling the model not to guess is enough.
Fact
Instructions lower the rate but do not remove it. Combine the instruction with a retrieval threshold, a citation check and an evaluation set of unanswerable questions.
Myth
A refusal means the system failed.
Fact
A correct refusal is a success. A confident wrong answer is the failure. Track refusals, because many refusals on answerable questions point to a retrieval problem.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Build a test set that includes unanswerable questions and measure two rates: answerable questions answered correctly, and unanswerable ones refused. Tuning one without the other overfits.
- ▸
Log refusals with the question and the best retrieval score. Clusters of refused questions are the best list of missing documents.
- ▸
Citations can be requested in structured output (an
answerfield and asourcesarray of numbers), which is easier to validate than parsing brackets from prose (Phase 4). - ▸
Never display a link or title that came from the model. Display the title and link from your own metadata for the verified source number.
Remember this
- 1
Why this exists. Retrieval does not stop a model from making things up: it may ignore the sources, mix two of them, or answer a question the sources do not cover from its own memory, in the same confident voice. Users cannot tell. Citations make an answer checkable, and a refusal path makes missing knowledge visible instead of dressed up as an answer.
- 2
Numbered sources. The prompt lists the retrieved chunks as
[1],[2]and tells the model to cite them like that after each claim. Short numbers are easy for the model to copy exactly, easy for you to parse, and easy to map back to a document name, a page and a link for the reader. Do not ask the model to cite titles or URLs from memory: it will invent plausible ones. - 3
Check the citations in code. After the model answers, parse the
[n]markers and verify: (1) there is at least one (unless the answer is a refusal), (2) everynis between 1 and the number of sources you actually sent, and (3) the claim is supported. Full support checking needs a person or a judging model (Topic 9.4), but a cheap and surprisingly useful test is: every number in the answer must appear in the cited source. The first example catches a made-up 60-day window, a citation of source [3] when only two exist, an uncited claim, and a right sentence attached to the wrong source. - 4
The refusal path. Give the model one fixed sentence to use when the sources do not answer the question (the example uses
Not in the documents.). A fixed sentence is easy to detect, to count, to translate in the interface and to route (to a human, a search page, a feedback button). Tell the model exactly when to use it in the system message, and test it with questions that have no answer in your documents. - 5
Refuse before the model, too. If retrieval finds nothing above the minimum score you can skip the model call and reply with the refusal directly: cheaper, faster and safe. The second example shows why a score is a blunt tool: "do you sell locks" and "what is the weather today" score 0 and are refused, but "how much is a lock" scores 0.7071 because the fake embedder sees
muchand thinks of cost, so a threshold of 0.30 lets it through and it would be answered from the shipping document. Raising the threshold to 0.80 fixes that one but also refuses "how much does it cost". That is a trade-off to tune on real questions, which is why the model-side refusal and the citation check stay in place as the second and third guard. - 6
What users see. Show the cited source next to the answer (document name, link, a short quote). It builds trust, lets people verify, and gives you feedback signals: a cited source that the user marks as unhelpful points at a retrieval or document problem. Never show a citation your code could not verify.
Explain it without notes
Why do citations improve trust, and what must your code do with them?
Why is a fixed refusal sentence better than letting the model phrase its own?
Why can a similarity threshold not fully solve the refusal problem?
How would you test a RAG system's refusal behaviour?
Practice
Write citations(text) that returns the list of integers found as [n] in a text, in order. Print it for "A [1] and B [2], again [1].".
Write valid_citations(text, n_sources) that returns True only if the text has at least one citation and every cited number is between 1 and n_sources. Print it for three texts.
Trade-offs
- ↔
Strict checks reject more good answers but keep wrong ones out; loose checks do the reverse.
- ↔
Refusing early saves model calls but cannot see subtle cases; a model-side refusal costs a call but uses the sources.
- ↔
Citations add a few tokens to every answer and need a parser, in exchange for verifiability.
Done when you can
Done when you can number sources in a prompt and parse the [n] markers from a reply.
Done when you can reject citations to sources that were not provided and numbers that are not in the cited text.
Done when you can give the model a fixed refusal sentence and detect it.
Done when you can explain the trade-off of a retrieval threshold and test refusals with unanswerable questions.