Topic 10.7
Poisoned and Untrusted Sources
In one line
Anyone who can add a document can change the answers. Decide who may add what, record where every passage came from and how far it is trusted, and check that stored vectors still match their text.
Think of it like this
A village noticeboard anyone can pin notes to. Most notes are fine. One day a note appears saying the bakery has moved, in the same handwriting and style as the real ones, and people walk to the wrong street. The fix is not a better noticeboard. It is knowing who pinned each note, and giving the council's notes more weight than a stranger's.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Data poisoning
- Adding wrong or misleading documents on purpose (or by accident) so the system gives wrong answers.
- Provenance
- The recorded origin and history of a passage: source, file, version, who added it, when.
- Allow-list
- A list of the only sources allowed in. Everything else is refused by default.
- Trust level
- A label on a source saying how far its content may be believed, such as reviewed or untrusted.
- Integrity check
- A test that stored data still matches what it should be, such as a vector matching its text.
- Quarantine
- Holding an item out of use, but not deleting it, until someone has reviewed it.
- Golden questions
- A fixed set of questions with known right answers, run after every change to catch regressions.
Step by step
01Step 1: list the sources and decide which are allowed
Write down every route by which text can enter. Put reviewed internal sources on the allow-list with the trust level reviewed. Customer uploads can be allowed as untrusted. Everything else is rejected.
02Step 2: attach provenance at ingestion
Each chunk carries source, trust, topic and a content hash. The example's ingestion lines print the decision for each candidate: D1, D8 and U1 are accepted, and W1 is rejected.
03Step 3: retrieve, then group by topic
For a refund question both D1 (reviewed) and U1 (untrusted) are retrieved. Grouping the hits by topic exposes that they disagree. The pass topic has no hits for this question, so the example prints "nothing retrieved" for it: no hits, no conflict.
04Step 4: let trust break the tie, and say so
The more trusted passage is used, the other is set aside, and the conflict is logged for a human. The answer is never produced from an untrusted passage when a reviewed one covers the same topic.
05Step 5: check the vectors
Re-embed each text and compare with the stored vector. The swapped vector for U1 ranks first before the check and is gone after it.
06Step 6: run the golden questions after every ingestion
A fixed set of questions with known answers (Phase 9) is the cheapest alarm for poisoning: if the refund answer changes after a batch of uploads, the batch is suspect.
Try it yourself
- 1
Allow a new source
Add
"web-crawl": "untrusted"to the allow-list. What happens to W1? Is it now harmful for the pass question? Which part of the example decides what is used? - 2
Tie on trust
Change U1's source to
"support-wiki". Both are now reviewed. Which passage is used, and is that a good rule? What other signals (a date, an owner) would you add? - 3
Corrupt a different vector
In the second example swap the vector of D4 for a copy of D2's. Does the check find it? Does the ranking change for the pass question?
Code & diagrams
The "topic" field is set by ingestion here to keep the example short. In practice it comes from metadata, a classifier or a human tag.
Expected output
ingest D1: accepted as reviewed
ingest D8: accepted as reviewed
ingest U1: accepted as untrusted
ingest W1: rejected (web-crawl is not on the allow-list)
Q: Is a cancelled ride refunded?
topic refund: conflict, using D1 (reviewed), set aside U1 (untrusted)
topic pass: nothing retrievedThe embedder counts four letters, which is a toy. With a real model, vectors from the same text can differ slightly between runs or versions, so compare within a tolerance, and pin the model version.
Expected output
before the check: U1=165, D2=60, D8=55
integrity check: mismatch for U1
after quarantine: D2=60, D8=55, D4=49Try: Press Search. The planted passage (promising refunds at any time) is not in the top 1: it scores higher than the real refund passage. Change the question to one that uses the real passage's own words, such as "Can I get a refund if I cancel at least 2 hours before the start?", and the real passage wins (about 0.64 against 0.52). Ranking cannot tell which of the two is true, which is why provenance and trust levels exist.
Goals
- The real refund rule ranks first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust everything the same
In the Python tab of the first example, change "customer-upload": "untrusted" to "customer-upload": "reviewed" in SOURCES, and change the question to "Is my cancelled ride refunded in full at any time?" (words that match the uploaded note closely). Run.
Myth vs fact
Myth
Only outside attackers poison data.
Fact
Most bad documents are honest mistakes: an old draft, a wrong copy, a test upload. Provenance and trust levels help with both.
Myth
Vectors are safe because nobody can read them.
Fact
Not being readable is the problem: a swapped vector looks like any other. Check them against their text.
Myth
The model will notice a source that looks wrong.
Fact
A model has no way to know which of two confident passages is true. Decide trust in code, before the prompt is built.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Add time and ownership to trust. A reviewed passage that has not been updated in three years can be less right than a new one. Store
ownerandlast_reviewed, and prefer fresh over stale within the same trust level (Topic 8.5, Topic 11.2). - ▸
Hash every chunk at ingestion and store the hash with the vector. A verification job can then detect text or vector changes made outside the pipeline without re-running the embedding model for every row: compare the stored hash with the hash of the stored text, and sample a few rows for full re-embedding.
- ▸
Treat a new high-ranking source as an alert. If a document uploaded yesterday is now the top hit for the 20 most common questions, someone should look at it before users act on it.
- ▸
Model-based poisoning is also possible at the embedding step: if you use a model service, pin its version and verify what you send. Do not accept an embedding vector supplied by a client, only text, and embed on your side.
Remember this
- 1
Why it is needed. Topic 10.2 was about a passage giving the model orders. This is quieter: a passage that gives wrong facts. Retrieval ranks by closeness to the question, not by truth, so a planted passage that repeats the question's wording can outrank the real one. The search lab shows it: a made-up passage promising a full refund at any time scores about 0.64 against the question, higher than the real refund passage at about 0.53. The model then answers confidently from the wrong source. This is data poisoning by an author, or by a careless upload.
- 2
Control who may write. Every route that adds documents is a way to change the answers. Keep an allow-list of sources: the handbook repository, the support wiki, an upload form for customers. Anything else (a web crawl nobody reviewed, a new connector) is rejected until someone decides it should be trusted. In the example the crawled page W1, which claims the pass is free, is refused at ingestion and never reaches the index.
- 3
Record provenance. Provenance is the history of a passage: which source, which file, which version, who added it and when. Store it as metadata on every chunk (Topic 4.6). When an answer is wrong you can then ask where the passage came from, and when a source turns out to be bad you can remove everything it added with one query.
- 4
Give sources a trust level, and use it. The example has two levels: reviewed (the handbook and the wiki, changed through review) and untrusted (customer uploads). When two retrieved passages on the same topic disagree, the system picks the more trusted one, says that it did, and sets the other aside for a human to look at. Here the handbook's 2-hour rule beats an uploaded note promising refunds at any time. Topic 8.5 covers disagreeing sources in answers; this adds trust as one of the tie-breakers, together with dates and authority.
- 5
Vectors need the same care as text. A stored vector is trusted by search but cannot be read by a person. If someone with write access to the store swaps in a vector, a harmless-looking passage can rank first for questions it has nothing to do with. The second example swaps the vector of an unrelated sentence for a scaled copy of the pass passage's vector and it takes first place. A routine integrity check re-embeds the text and compares it with the stored vector (with a real model, compare within a small tolerance, or on a random sample if the corpus is large) and quarantines mismatches.
- 6
Lock down the store, log the writes, review the surprises. Give the application's read path a read-only database role, and only the ingestion job the right to write. Log every write with the source and a content hash. Alert on unusual changes: many documents from one source in a day, a document that suddenly outranks the handbook for common questions, a vector that fails its check. Keep a small set of golden questions (Topic 9.2) and run them after each ingestion: a poisoned passage often shows up as a changed answer.
Explain it without notes
What is data poisoning in a RAG system, and why does ranking not protect against it?
What do an allow-list, provenance and trust levels each contribute?
How can a vector in the store be wrong, and how do you detect it?
Two retrieved passages disagree. What does your system do?
Practice
Write pick(hits) where each hit is (doc_id, trust) and trust is "reviewed" or "untrusted". Return the id of the first reviewed hit, or of the first hit if none is reviewed, or None for an empty list.
Write content_hash(text) using h = (h * 31 + ord(char)) % 1000003 starting from 7, and tampered(rows) where rows maps id to (text, stored_hash) and returns the ids whose stored hash differs.
Trade-offs
- ↔
A tight allow-list: fewer surprises and slower onboarding of new sources. Add sources deliberately.
- ↔
Preferring reviewed over untrusted: protects against bad uploads and can hide a correct customer-supplied fact. Review the set-aside queue.
- ↔
Full re-embedding checks: strong detection and cost per row. Use hashes everywhere and full checks on a sample.
Done when you can
Done when you can name every route into your index and say which are allowed.
Done when you can show provenance and trust fields on a chunk and use them in retrieval.
Done when you can explain how a swapped vector is detected.
Done when you can describe the golden-question check after each ingestion.