Topic 8.5
When Sources Disagree
In one line
Real handbooks hold old and new versions of the same page, and near-duplicate passages score almost the same. Dates, status and authority must travel with each passage into the prompt, and a genuine conflict is shown, not hidden.
Think of it like this
Two copies of the staff rota pinned to the notice board. One is from last year. If you grab whichever is on top, you might turn up on the wrong day. The fix is a date and the word 'current' on each, and the habit of looking at the date. A model has no eyes on the notice board: it only knows what the prompt says.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Superseded
- Replaced by a newer version and no longer in force.
- Effective date
- The date from which a document's rules apply.
- Status
- A document's state: current, superseded, draft.
- Near-duplicate
- Two passages that say almost the same thing, so they score almost the same.
- Authority
- How much a source should be trusted: an official policy over a draft over a chat message.
- Conflict
- Two sources that give different answers to the same question.
- Metadata filter
- A rule on stored fields (status, date) that removes chunks before ranking.
Step by step
01Step 1: the old price list scores higher
Open the lab below. For 'What does the monthly pass cost?' both versions are in the top two, and the old one can come first. Similar text means similar score.
02Step 2: store topic, date and status with each chunk
The example's documents have (id, topic, title, date, text). A topic is the label that groups versions of one page.
03Step 3: label versions
For each topic in the hits, the newest effective date is current and the others are superseded. Ties in date are a data bug: report them.
04Step 4: put the labels in the prompt
Each header now shows effective DATE, current or superseded, and the instruction says what to do with the difference.
05Step 5: choose filter, label or show
If nobody needs history, filtering is simplest and cheapest. If questions can ask 'what was it before?', keep and label. If two current sources disagree, show both.
06Step 6: test it with the conflict in the golden set
Add 'What does the monthly pass cost?' to your tests with the expectation '25 dollars, current source cited'. A prompt change that flips the answer to 20 fails the test.
Try it yourself
- 1
Flip the dates
In the Python tab, swap the two dates so the old list looks newer. Run it. Whose price does the answer give? Notice that the code did what the data said.
- 2
Make a tie
Give both passages the same date. What does
label_versionsdo? Decide what your system should do with a tie and add that branch. - 3
Filter instead
Change the code to drop superseded hits before building the prompt. Compare the two prompts and say what the reader loses and gains.
Code & diagrams
The answer is scripted: a real model follows the prompt's rule only most of the time, so check that the current source is cited.
Expected output
topics with more than one version: monthly-pass
[1] Monthly pass (2024 price list) (effective 2024-03-01, superseded)
The monthly pass costs 20 dollars and includes unlimited rides of up to 30 minutes.
[2] Monthly pass (effective 2025-06-01, current)
The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.
[3] Refunds (effective 2025-01-10, current)
A ride cancelled at least 2 hours before its start time is refunded in full. Later cancellations get ride credit instead.
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [2] An older version [1] from 2024-03-01 said: the pass costs 20 dollars and includes unlimited rides of up to 30 minutes
drop-the-old-ones option keeps: D8, D1Try: Search for 'What does the monthly pass cost?' and read both pass passages' Cosine. Open the Similarity grid and find how similar the two pass passages are to each other. Then ask 'How much was the pass in 2024?' and see whether retrieval alone can tell which version you meant.
Goals
- The current price (25 dollars) is retrieved
- The old price (20 dollars) is retrieved too
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust the oldest date
In the Python tab, change date > newest[topic] to date < newest[topic] in label_versions.
Myth vs fact
Myth
Embedding search will find the newest version.
Fact
It finds the most similar text. Old and new versions of one page are nearly identical, and the old one can score higher.
Myth
Tell the model to use the newest source and it will.
Fact
Only if the prompt shows the dates and the status. The model sees text, not the document store.
Myth
Always prefer the newest document.
Fact
A draft, or a document that is not yet effective, is not policy. Rank by status first, then date.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prevent the problem at ingestion: when a new version of a page arrives, mark the previous version superseded in the same step (Topic 4.7), instead of finding out at query time.
- ▸
A conflict between two different pages (not versions of one) needs a different detector: a model judge or an entailment model comparing claims. It is slower and imperfect, so run it on the final answer, not on all pairs of chunks (Topic 8.7).
- ▸
Show dates in the interface next to citations ('policy effective 2025-06-01'). Readers calibrate trust from it, and support staff can spot a stale answer at a glance (Topic 11.2).
Remember this
- 1
Why this happens. Companies keep old price lists, drafts and archived policies next to the current ones. An embedding model sees two pages about the monthly pass and scores them almost the same. In the lab, the 2024 price list and the current price passage have a cosine similarity of about 0.96 to each other, and for 'What does the monthly pass cost?' the old one scores slightly higher (about 0.76 against 0.75). Retrieval cannot tell which is right. It returns both.
- 2
The model cannot tell either. If both passages go into the prompt without dates, the model sees two facts that disagree: 20 dollars and 25 dollars. Whichever comes first, or sounds more confident, tends to win. Without metadata a correct answer would be luck.
- 3
This phase adds a document. The handbook gains D16, 'Monthly pass (2024 price list)': 20 dollars and rides up to 30 minutes, effective 2024-03-01. D8 (25 dollars, 45 minutes) now carries an effective date of 2025-06-01. Both passages are used in the example. Later phases may add more documents, but never change these texts.
- 4
Metadata is the fix. Each chunk carries
topic,effective dateandstatus(current, superseded, draft). Your code can then (a) filter superseded documents out before retrieval, which is the cleanest option when nobody asks about history, (b) keep them and label them in the prompt, so the model can answer 'what was the price in 2024?', or (c) show the conflict. Filtering at retrieval time is covered with metadata filters in Topic 5.5, and the fields themselves in Topic 4.6. - 5
Label the prompt. The prompt's source header gains the date and the status:
[2] Monthly pass (effective 2025-06-01, current). The instruction adds a rule: 'If sources disagree, prefer the current one and say what the older one said.' The example detects the conflict in code (two sources on one topic) and builds exactly that prompt. - 6
Authority and dates. Newer is not always right: a draft dated tomorrow is not policy, and a forum post is not the handbook. Rank by status first (official, draft, discussion), then by effective date. Keep 'published' and 'effective' dates separate: a price change announced in May and effective in June means two different dates.
- 7
When no one wins, say so. If two *current* documents disagree (two shops publishing different hours), do not pick. Return both with their sources and a plain sentence: 'The handbook gives two different answers.' A visible conflict gets fixed; a silently chosen one hides the bug.
Explain it without notes
Why can an old version of a page outrank the current one in retrieval?
What must the prompt contain for the model to handle versions correctly?
Name three ways to handle superseded documents and when to use each.
What should the system do when two current documents disagree?
Practice
Write label(hits) where hits is a list of (doc_id, topic, date) and the result is a list of (doc_id, status): current for the newest date per topic, superseded for the rest. Print the result for D16 (monthly-pass, 2024-03-01), D8 (monthly-pass, 2025-06-01), D1 (refunds, 2025-01-10).
Write has_tie(hits) returning True when two hits of the same topic have the same newest date. Print the result for the hits above, and for the same hits with D16's date changed to 2025-06-01.
Trade-offs
- ↔
Filtering superseded documents: clean and cheap, but history questions fail.
- ↔
Labelling in the prompt: keeps history, but costs tokens and relies on the model following the rule.
- ↔
Showing the conflict: honest and fixes data, but the answer is less crisp for the user.
Done when you can
Done when you can explain why retrieval cannot pick the newest version.
Done when you can attach topic, date and status to chunks and label versions in code.
Done when you can put dates and status in the prompt with a rule for differences.
Done when you can say what to do when two current sources disagree.