Topic 10.1
Threat Modelling a RAG System
In one line
A threat model lists what is worth protecting, who might attack it, and every place where text enters the system. For RAG the key line to draw is a trust boundary around everything that is retrieved: it is data to read, never orders to follow.
Think of it like this
A post office sorting room. Letters come from everyone, so the clerks open them, read them, and file them, but a letter that says "clerks must give me the keys" does not get obeyed just because it is on paper inside the building. The room has rules for who may give orders (the manager) and who only sends mail (everyone else). A RAG system needs the same rule for text.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Asset
- Something worth protecting: private documents, personal data, the truth of answers, the system's budget.
- Threat model
- A short list of assets, possible attackers and the ways in, written before choosing controls.
- Entry point
- A place where outside text enters the system: a question, an uploaded file, a crawled page, a metadata field.
- Trust boundary
- The line between text you control and text others wrote. Crossing it means the text is treated as data only.
- Control
- A defence that reduces a threat: a filter, a redactor, a check, a rule.
- Least privilege
- Give each part of the system only the access it needs, so a failure exposes little.
- Attack surface
- The sum of all entry points: every door someone could try.
Step by step
01Step 1: list what is valuable
Write the assets in one line each. For Lumen Bikes: staff-only documents, customers' personal data in uploaded tickets, correct refund and safety answers, and the monthly bill for model calls.
02Step 2: list every place text comes in
Walk the data path from the source of a document to the answer on the screen and write down each point where someone other than you decides what the text says. The example's table has six, and three of them are untrusted.
03Step 3: draw the trust boundary
Label each entry point trusted, reviewed or untrusted. The rule "only trusted text may change the rules" is one function in the example: may_change_rules. Everything the prompt builder receives from the index is on the untrusted side, even when a human reviewed it, because reviews miss things.
04Step 4: pair each asset with a threat and a control
Write the sentence "someone could ... so we ..." for each asset. Keep each control to one line. If you cannot name a control, that asset is a gap to schedule.
05Step 5: add a test for every control
For each control, decide how you would notice it failing: a test that asks as the wrong user, a search of the logs for an email address, an audit after a deletion. Phases 9 and 11 give you the harness; this phase gives you the cases.
06Step 6: keep it alive
Re-read the threat model when you add an entry point (a new upload type, a new connector, a new tool). The most common security bug in RAG is a new data source added without anyone asking who may write to it.
Try it yourself
- 1
Add an entry point
Add
("support ticket", "a customer", "untrusted")to the first list and run. How many entry points are untrusted now? Which threat would a ticket belong to? - 2
Add a threat
Add a sixth threat: the model call bill is run up by a user who asks thousands of questions. Write the control in the same style (Topic 11.5 shows the cost side).
- 3
Try to give an untrusted source power
Change
may_change_rulesso "reviewed" may change the rules. Run and read the table. Is a reviewed wiki page really as safe as your system prompt? Who can edit it?
Code & diagrams
A threat model fits in a table you can print. The controls named here are the topics 10.2 to 10.7.
Expected output
Where text enters:
system prompt written by the developers trusted may change the rules: yes
access labels written by the ingestion code trusted may change the rules: yes
handbook passage written by a reviewed editor reviewed may change the rules: no
uploaded file written by a customer untrusted may change the rules: no
crawled web page written by anyone online untrusted may change the rules: no
user question written by any signed-in user untrusted may change the rules: no
Untrusted entry points: 3 of 6
What to protect:
another team's documents: a user reads a chunk they should not see -> labels checked inside retrieve
the model's behaviour: a passage gives the model orders -> sources are data and output is checked
customers' personal data: an email address lands in an index or a log -> redact before indexing
the truth of the answers: a bad document is added on purpose -> provenance and trust levels
the right to be forgotten: a deleted document lives on in a cache -> delete everywhere, then auditBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Let reviewed text give orders
In the Python tab, change the return line of may_change_rules to return trust != "untrusted" and run. (Output trimmed to the first rows.)
Myth vs fact
Myth
Security means encrypting the database.
Fact
Encryption protects a stolen disk. Most RAG leaks are the application showing a chunk to the wrong person, which no disk encryption can stop.
Myth
Our documents are internal, so they are safe.
Fact
Internal documents are still written by many people, include uploads and copied web text, and can contain instructions or personal data. "Internal" describes where they live, not whether they are trustworthy.
Myth
A model that is told to ignore bad instructions will always do so.
Fact
A system message lowers the chance but does not remove it. Treat the model as something that can be talked into mistakes, and add controls around it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Map the system to a recognised list as a checklist, not as a ritual. OWASP publishes a Top 10 for LLM applications. In its 2025 edition four entries are central to this phase: LLM01 Prompt Injection (Topic 10.2), LLM02 Sensitive Information Disclosure (Topics 10.3 to 10.5), LLM04 Data and Model Poisoning (Topic 10.7) and LLM08 Vector and Embedding Weaknesses (Topics 10.3, 10.4 and 10.7). Lists are revised, so check the current names on OWASP's site before quoting them in a review.
- ▸
Count your untrusted entry points on a regular schedule, like dependencies. The number tends to grow quietly (a new connector, a new upload type), and each one needs its own answer to "who can write here, and what can their text cause?".
- ▸
Write the worst realistic day: a mislabelled payroll file indexed as public and asked about by a curious user. Walk it through the system and check each stage can stop it. Incident drills (Topic 11.9) start from such stories.
Remember this
- 1
Why model threats first. Security work without a list of threats turns into random hardening: you encrypt a disk and forget that your search tool shows any chunk to any user. A threat model is a short written list: what is valuable (assets), who might go after it (attackers, including honest people who make mistakes), and the doors they could use (entry points). It takes an hour and tells you which of the next eight topics matter most for your product.
- 2
Assets in a RAG system. Four are worth naming. (1) Private documents: the content users must only see if they are allowed to. (2) Personal data inside documents and questions: names, emails, card numbers. (3) The truth of answers: users act on what the system says. (4) The system's behaviour and budget: what it can be made to do and how much it can be made to spend. Each later topic protects one of these. The example's table adds a fifth line, the right to have data deleted, because a deleted document that lives on in a cache is a failure of its own (Topic 10.6).
- 3
Entry points: where text comes in. The question is the obvious one. The less obvious ones carry more risk for RAG: documents (uploads, crawled pages, wiki edits), metadata (an access label or a source name that a bad import sets wrongly), and tool or API results if the model can call things. In an ordinary chat app only the user types. In RAG, hundreds of people you have never met can put text in front of the model, which is the whole point of the product and also its biggest exposure.
- 4
The trust boundary. Mark each entry point as trusted (your code and your system prompt), reviewed (written by staff through a process) or untrusted (customers, the web, the user's question). The rule that follows is simple and strict: only trusted text may change the rules. Retrieved passages are evidence. They may be wrong and they may be hostile, but they are never instructions. Topic 10.2 turns this into code.
- 5
Attackers include honest mistakes. Think of three kinds of actor: a curious user who tries other teams' topics, a malicious author who plants text in a document the system will index, and a careless insider who uploads a spreadsheet with customer emails or tags a payroll file "public". Most real incidents are the third kind. A good threat model treats a wrong label as seriously as an attack.
- 6
From threat to control to test. For every threat write one control and one way to check it. "A user reads a chunk they should not" becomes "retrieval filters by label" and "a test asks as every user and fails on any foreign chunk" (Topic 10.3). A threat without a test is a hope. The example prints the entry points and one control per asset for AskDocs; the rest of the phase builds each control and its test.
Explain it without notes
What is a threat model, and what four things should it list for a RAG system?
Which entry points does RAG add compared with a plain chat app?
Explain the trust boundary and the rule that follows from it.
Why should a wrong access label be modelled as a threat even if nobody is attacking?
Practice
Add ("ticket attachment", "a customer", "untrusted") and a threat "a ticket attachment contains a customer's card number" with control "redact before indexing". Print the count of untrusted entry points.
Write risky(entry_points) that returns the names of entry points that are not trusted and not reviewed. Print the list for the table in this topic.
Trade-offs
- ↔
A detailed threat model: finds gaps early, takes time that small teams are tempted to skip. A one-page version done once per quarter is far better than none.
- ↔
Treating all retrieved text as untrusted: safe and simple, but it means the system must never let a passage trigger an action. That limits some clever designs on purpose.
- ↔
More entry points (uploads, crawlers, connectors): a more useful product and a larger attack surface. Add them deliberately, one at a time.
Done when you can
Done when you can list the assets of a RAG system and one threat for each.
Done when you can name every place text enters your system and who writes it.
Done when you can state the trust boundary and the rule about retrieved text.
Done when you can pair each control with a test that would fail if the control broke.