Topic 8.3
Citations You Can Check
In one line
A citation is a claim about the evidence, and claims can be wrong. Parse the markers in the answer, check each against the numbers in the prompt, report unused and invalid ones, and show the reader the real source behind every marker.
Think of it like this
A student's essay with footnotes. A footnote number tells you where to look, but a teacher still checks that footnote 4 exists, that it points to a real book, and that the book says what the essay claims. A footnote the essay cites but the bibliography lacks is an instant red flag. Your code is the teacher.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Citation
- A marker in an answer that points to the source it relies on.
- Valid citation
- One whose number exists among this request's sources.
- Invalid citation
- A marker for a source number that was not in the prompt.
- Unused source
- A source in the prompt that the answer never cited.
- Provenance
- Where a piece of information came from: document, version and place.
- Verifier
- Code that checks a model's output against rules, without trusting the model.
Step by step
01Step 1: keep the number-to-document map
When you build the prompt, keep a dictionary from source number to document id. The prompt is generated from the same list, so they cannot drift apart.
02Step 2: find the markers
Collect every [n] in the answer as a sorted set of integers. The example uses a regular expression in Python, JavaScript and Java.
import re
cited = sorted({int(n) for n in re.findall(r"\[(\d+)\]", answer)})03Step 3: compare with the sources
Invalid = cited numbers missing from the map. Unused = map numbers never cited. A good answer cites at least one source and has no invalid ones.
04Step 4: four answers
Answer A cites source 1: ok, and sources 2 and 3 are unused. Answer B cites 2 and 4 but there are only three sources: invalid. Answer C cites nothing: fail. Answer D cites 3 and 2: ok.
05Step 5: decide what a failure does
Choose a rule and apply it every time: retry once with a stricter instruction, remove the bad marker and flag the answer, or fall back to the refusal. Log the failure either way.
06Step 6: show the reader the source
Render each valid marker as a link to the document and the passage. The reader can then verify without trusting the model or you.
Try it yourself
- 1
Add a sneaky answer
Add an answer E with
[1 ](a space inside) and another F with(1). Which does the parser accept? Decide which forms your instruction allows and make the parser strict about them. - 2
Ignore brackets that are not citations
Add an answer G that says
The sign said [sic] 25 dollars [1].Check that[sic]is not counted as a citation. - 3
Log unused sources
Print the unused sources for every answer on one line, then think: when would a high unused count be a warning sign?
Code & diagrams
A valid marker only proves the source was in the prompt. Topic 8.7 checks whether the source supports the sentence.
Expected output
A: cited 1 | invalid none | unused 2,3 | ok
B: cited 2,4 | invalid 4 | unused 1,3 | FAIL
C: cited none | invalid none | unused 1,2,3 | FAIL
D: cited 2,3 | invalid none | unused 1 | okTry: Press Search, then Answer. The model gets the top passages as [1], [2], [3] and must cite them. Check that the marker it uses points at the passage that really holds the answer, and that both goals pass. Then change the question to one the passages only half answer ('Can I cancel a ride 30 minutes before it starts?') and see what it cites.
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Answer from the sources · real model
Search first, then sign in (free) to send the retrieved passages to a real model.
- Cites source [1]
- Does not refuse
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust the numbers
In the Python tab, replace the line that builds invalid with invalid = [].
Myth vs fact
Myth
If every citation is valid, the answer is correct.
Fact
Valid means the source was in the prompt. Whether it supports the sentence needs the claim checks in Topic 8.7.
Myth
Models never invent citations when given sources.
Fact
They do sometimes cite numbers outside the list or cite none. A parser is cheaper than finding out from a user.
Myth
Citations at the end of an answer are enough.
Fact
A marker per claim lets a reader and your checks match each statement to evidence. A pile at the end proves little.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the cited numbers, the invalid ones and the unused ones for every answer. The invalid rate over time is a cheap health metric that reacts to prompt or model changes before users complain (Topic 11.6).
- ▸
If you stream the answer, a marker can arrive split across chunks (
[then1]). Topic 8.6 shows how to hold back text so a half marker never reaches the screen. - ▸
Store the exact passages shown with the answer, not only the ids. If a document is edited tomorrow, the reader can still see what the answer was based on, which matters for audits.
Remember this
- 1
Why check citations in code. Models sometimes cite a source number that was never in the prompt, cite none at all, or attach a real number to a claim that source does not support. Showing a reader a citation that looks like proof is worse than showing none, because it earns trust it has not earned. The check costs a few lines and no model call. LLM Engineering Topic 8.6 gave the short version; this topic goes further.
- 2
Parse the markers. An answer cites with
[1],[2]. Scan the text for square brackets with whole numbers inside, collect the unique numbers and sort them. A plain regular expression or a small scanner both work, and the parser must ignore brackets that are not citations (for example[sic]). - 3
Three checks on the numbers. Valid: every cited number is between 1 and the number of sources in this request. Present: an answer that is not a refusal has at least one citation. Unused: sources that were in the prompt and never cited are not an error, but a long list of them can mean the model ignored its evidence, and they are worth logging.
- 4
What the numbers cannot tell you. A valid marker proves the source existed in the prompt. It does not prove the source supports the sentence. Topic 8.7 adds checks on the claim itself (word overlap, numbers, a judge model). Treat citation checks as the cheap first layer.
- 5
Show the source to the reader. Your code, not the model, maps each marker to a document id, title and a link, and shows the passage itself on click. A reader who can open source 2 can check the answer without trusting you. If a marker fails validation, do not show the answer as if it were fine: retry, drop the bad citation and flag it, or fall back to a refusal.
- 6
Where to place markers. Ask for a marker at the end of each sentence or claim it supports, not one pile at the end of the answer. Marker-per-claim is what lets the reader and your own checks match evidence to statement.
Explain it without notes
Name the checks you would run on an answer's citations without calling a model.
What does a valid citation prove, and what does it not?
Why should your code, not the model, map a marker to the document shown to the reader?
What should happen when the check fails?
Practice
Write check_citations(answer, n_sources) returning (cited, invalid), both sorted lists of ints, using re.findall. Print the result for "Pass [1], late fee [3], refund [3]" with 2 sources.
Write strip_invalid(answer, n_sources) that removes any [n] marker whose number is not between 1 and n_sources, keeping valid ones. Print the result for "Pass costs 25 [1] and rides last 45 [5]." with 2 sources.
Trade-offs
- ↔
Strict citation formats: easy to parse and verify, but the model sometimes strays, so you need retries.
- ↔
Marker per claim: best for verification, but longer, busier answers.
- ↔
Stripping a bad marker: keeps the answer, but leaves a claim with no support. Retry or refuse is safer for important answers.
Done when you can
Done when you can parse citation markers from an answer.
Done when you can report invalid, missing and unused citations.
Done when you can explain what a valid citation does and does not prove.
Done when you can say what your system does on a failed check and log it.