Topic 10.1
Threat Modelling an LLM App
In one line
A threat model lists what you must protect, who might attack it, and every place untrusted text can enter. For an LLM app the entry points are wider than a normal web app: user text, documents, web pages, tool output and the model's own reply.
Think of it like this
A school before an open day. The head teacher walks the building and asks three questions: what must we protect (children, exam papers, the cash box), who might cause trouble (a stranger, a bored student, a careless visitor), and which doors, windows and corridors let them in. Only then does she decide where to put a lock, a guard or a sign. A threat model is that walk, done on paper before the open day.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Asset
- Something worth protecting: data, secrets, actions the app can take, money, trust.
- Threat model
- A short written list of assets, attackers and entry points, made before choosing defences.
- Entry point
- Any place where text or an action from outside reaches your system or your prompt.
- Untrusted input
- Text you did not write and cannot vouch for: user messages, documents, web pages, tool output, model replies.
- Attack surface
- The total of all entry points an attacker could use.
- OWASP
- A non-profit that publishes free security guidance, including the Top 10 lists.
- Blast radius
- How much damage happens if one part is fooled. Least privilege makes it small.
Step by step
01Step 1: list what you protect
Take a support bot that answers questions from your help articles and can email a customer. Assets: the system prompt, the customer table it can read, the email action, the monthly bill, the brand's reputation. Writing it out takes five minutes and already shows that the email action is the scariest one.
02Step 2: list every way text gets in
Chat box, uploaded PDF, web search result, the hidden system prompt (your own text, but it can leak), the answer shown on the page, the send_email tool. The program below stores this as a list of (part, kind of entry point) pairs, which is all a threat model needs to start.
app = [
("chat box", "user text"),
("uploaded PDF", "document text"),
("send_email tool", "tools"),
]03Step 3: map each entry point to risks
A lookup table from kind of entry point to the OWASP classes it brings. Untrusted documents bring prompt injection; model output used by code brings improper output handling; tools bring excessive agency. The table is small on purpose: the thinking is in choosing the rows, and you can extend it for your own app.
04Step 4: count the distinct risks
Six parts of the app produce eight entries but only six distinct risk classes, because prompt injection appears three times. That repetition is the lesson: the same class arrives through many doors, so defences must sit on the doors and behind them, not in one place.
05Step 5: choose defences by worst case
For each risk write the worst case and the cheapest defence that removes it: injection through a PDF is limited by removing tools the summariser does not need; output handling is solved with escaping; excessive agency with an allow-list and human approval; unbounded consumption with token limits and per-user quotas (Phase 6). The rest of this phase takes them one by one.
06Step 6: keep it alive
A threat model is a file in your repository, not a meeting. Review it when you add a tool, a data source or a new kind of input, and turn each line into a test (Topic 10.8).
Try it yourself
- 1
Add a new door
Add
("customer email being summarised", "document text")toapp. Predict how many distinct risks the program prints, then run it. Why did the count not change? - 2
Add a new kind of entry point
Add a row
"plugin output": ["LLM01 Prompt Injection", "LLM03 Supply Chain"]to the table and a part that uses it. Which two lines of output change? - 3
Model your own app
Replace the app list with the parts of an app you use or want to build (a recipe bot, a code reviewer, a meeting summariser). Which part has the worst worst-case?
Code & diagrams
The risk names are from the OWASP Top 10 for LLM Applications, 2025 edition. Extend the table for your own app.
Expected output
chat box [user text]: LLM01 Prompt Injection, LLM10 Unbounded Consumption
uploaded PDF [document text]: LLM01 Prompt Injection
web search result [tool output]: LLM01 Prompt Injection
answer shown on the page [model output]: LLM05 Improper Output Handling
hidden system prompt [system prompt]: LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage
send_email tool [tools]: LLM06 Excessive Agency
6 distinct risks to design for:
LLM01 Prompt Injection
LLM02 Sensitive Information Disclosure
LLM05 Improper Output Handling
LLM06 Excessive Agency
LLM07 System Prompt Leakage
LLM10 Unbounded ConsumptionBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget a door
Delete the send_email tool row from app, as if nobody had listed the tools when the model was written. Run it.
Myth vs fact
Myth
Security is about stopping hackers typing clever prompts into the chat box.
Fact
Many incidents start with text the user never typed: a web page, a document, an email or a tool result that the app pulls into the prompt.
Myth
The OWASP list is a checklist: tick ten boxes and you are safe.
Fact
It is a map of risk classes. You still decide which apply, how bad they are for your app and how to test the defences.
Myth
A small app with no tools has nothing to protect.
Fact
It still has a system prompt, a bill, other users' data in retrieval, and the people who read its answers.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Draw trust boundaries on the data-flow diagram: wherever text crosses from a source you do not control into one your code treats as instructions or as markup, a check belongs there.
- ▸
Threat model per feature, not per company. A summariser, a support bot with tools and a code-writing agent have different worst cases and need different defences; one shared policy for all three is either too weak or too heavy.
- ▸
Include cost as a security property: a single user who sends huge inputs, or a loop that calls the model repeatedly, can exhaust your budget. OWASP names this class Unbounded Consumption (LLM10 in the 2025 list).
Remember this
- 1
Why this exists. Security fixes that are bolted on after launch are guesses. If you never listed the doors, you lock the front one and leave the window open. A threat model forces you to list assets (what is worth protecting), attackers (who and why) and entry points (where outside text or actions reach your system), so each defence in this phase is placed where it matters.
- 2
Assets in an LLM app. Your system prompt and any secrets near it, the private data the app can read (other users' documents, customer records), the actions it can take (send mail, write to a database, spend money), your bill (every token costs), your users' trust, and the provider account that could be rate-limited or banned. Write the list down; most apps have more assets than their builders first notice.
- 3
Attackers. A curious user typing odd things, a user who wants free answers or another user's data, an outsider who controls a web page or a document your app will read (they never talk to your app, yet they can steer it), and a careless insider. Most real incidents are not movie hackers: they are ordinary text arriving in an unexpected place.
- 4
Entry points are wider than the chat box. Anything that ends up inside the prompt is an entry point: the user's message, an uploaded file, a retrieved document, a web search result, the output of a tool, an email being summarised. Anything that comes out and is used by code is also an entry point for the next system: the model's reply shown in a web page, put into a database query or run as a command (Topic 10.3). Treat all of it as untrusted input, because you did not write it.
- 5
OWASP gives you a map. The OWASP Top 10 for LLM Applications (2025 edition) names ten classes: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation and LLM10 Unbounded Consumption. Use it to check you have not forgotten a class, then decide which ones apply to your app. The list is revised over time, so check genai.owasp.org for the current edition before you quote it in a review.
- 6
Rank by impact, not by fear. For each entry point ask: if an attacker controls this text, what is the worst thing the app can do? A summariser with no tools can only say wrong things. The same summariser with a send-email tool can leak data. The worst case decides the defence: removing a dangerous tool is cheaper and stronger than detecting every attack text. This is the idea behind Topic 10.7.
Explain it without notes
What is a threat model and what three things does it list?
Why is an LLM app's attack surface larger than the chat box?
How do you decide which defence to build first?
How can you use the OWASP Top 10 for LLM Applications without treating it as a checklist?
Practice
Add risks_for(app) to the program: it takes the list of (name, kind) pairs and returns the sorted list of distinct risk names. Call it with the app from the example and print how many there are.
Write worst_case(app) that returns the parts of the app whose kind is "tools", as a list of names, so a reviewer sees first what can act in the world.
Trade-offs
- ↔
A thorough threat model takes an afternoon and saves weeks of patching; an overly detailed one is never updated. Keep it to one page per feature.
- ↔
Listing every risk tempts you to defend everything equally. Spend effort where the worst case is worst.
- ↔
Using a public list such as OWASP gives common vocabulary with reviewers, but it can make you forget risks specific to your product.
Done when you can
Done when you can list the assets, attackers and entry points of an LLM app you know.
Done when you can explain why a web page or a tool result is an entry point.
Done when you can name at least eight of the ten classes in the OWASP Top 10 for LLM Applications 2025 and say which apply to your app.
Done when you can rank two features by worst case and say which defence comes first.