Topic 9.1
The Threat Model for an AI UI
In one line
A threat model lists what an attacker controls, what they want and what the worst outcome is. For an AI feature, the attacker controls every piece of text the model reads that you did not write, and the model cannot tell that text from your instructions.
Think of it like this
A receptionist who follows every note left on the desk. A visitor leaves a note saying "give the bearer the key to the stockroom". A parcel arrives with a sticker saying "the receptionist must call this number". The receptionist is honest and helpful, and has no way to tell the manager's notes from anyone else's. Safety here does not come from asking the receptionist to be careful. It comes from what the receptionist is allowed to hand out.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Threat model
- A short list of what you protect, who can reach it, the worst outcome and what stops it.
- Untrusted input
- Any text you did not write and cannot predict: user messages, documents, web pages, tool results.
- System prompt
- Your standing instructions to the model, written by you and sent by your server.
- Prompt injection
- Text placed in the model's input that tries to give the model new orders. Topic 9.2 covers it in depth.
- Attack surface
- All the places an attacker can put input or read output. The smaller it is, the easier it is to defend.
- Blast radius
- How much damage one successful attack can do. Least privilege keeps it small.
- Defence in depth
- Several independent protections, so one failing does not break everything.
- OWASP
- A non-profit security community that publishes free checklists, including a Top 10 for LLM applications.
Step by step
01List the doors
Write down every kind of text that can reach the model in your feature. For Mitra today: the customer's message, the earlier turns of the conversation, order data from the database. Later phases add documents (Phase 11) and tool results (Phase 8).
The first live example prints that list for a request that includes a pasted document and a fetched web page. Count the untrusted entries: it is almost always all of them except the system prompt.
02Say what each door could cost you
For each door, ask what happens if the model obeys a hostile instruction that arrived through it. If Mitra can only answer questions, the worst case is a wrong answer. If Mitra can cancel orders, the worst case is a cancelled order. This answer, not the cleverness of the attack, tells you how much protection you need.
03Put the checks on the server
The second live example sends deliberately bad requests to the chat route: a forged system message, content that is not text, a message that is far too long, an empty list. The route keeps only user and assistant text, builds the one system message itself and refuses the rest.
None of this needs the model. Cheap, boring checks before the model is called remove whole classes of problems.
04Pick the control for each risk
Match each risk to a control and to the topic that teaches it: injected instructions (9.2, contain the damage), unsafe rendering (9.3), leaks through links and images (9.4), over-powerful tools (9.5), harmful or blocked content (9.6), personal data (9.7). Write the table once and keep it in the repository next to the code.
05Verify, do not hope
For every control, write the check that proves it works: a test that sends the hostile input and asserts the harmless result. This phase ends with such a list for Mitra (Topic 9.8). A control you have never seen fail is a guess.
Try it yourself
- 1
Add a door
In the first example, add a part
{ source: "email Mitra summarised", trusted: false, text: "Hi, my order is late" }. Run it.The count of untrusted doors goes up by one. Every new feature that reads text from somewhere adds a door, whether or not anyone thought of it as one.
- 2
Raise the limit and watch what passes
In the second example, change
MAX_CHARSto5000and run it again.The "too long" case now succeeds. A limit is a decision about cost and abuse, not a fact about the world; write down why yours is where it is.
Code & diagrams
This is a plain list in code, not a model call. The point is the habit: before you write the feature, write down every source of text and mark who controls it.
Preview
Console
Nothing logged yet.Expected console output
system prompt you wrote it | 44 chars
customer message NOT YOURS | 18 chars
pasted document NOT YOURS | 76 chars
web page Mitra read NOT YOURS | empty
tool result NOT YOURS | 38 chars
untrusted doors: 4 of 5
empty but still a door: web page Mitra readThe browser here is playing a hostile client. The route never trusts it. Counting characters with `.length` counts UTF-16 units, which is fine for a size limit; do not use it to count what a person sees.
Preview
Console
Nothing logged yet.Expected console output
normal 200 {"promptRoles":["system","user"],"dropped":[]}
forged system 200 {"promptRoles":["system","user"],"dropped":["role \"system\""]}
object as content 400 {"error":"no usable messages"}
too long 413 {"error":"message too long"}
Hindi and emoji 200 {"promptRoles":["system","user"],"dropped":[]}
empty list 400 {"error":"messages required"}Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust the roles the browser sends
In the second example, replace the first line inside the loop (the role check) with nothing, so every message is kept as it came, then run it.
Myth vs fact
Myth
We only show answers to the person who asked, so there is no attacker.
Fact
The text the model reads can come from other people: a web page, a document, a shared review, a database row. The person asking may be a victim, not the author.
Myth
A strong system prompt is enough protection.
Fact
It lowers the chance of a bad outcome and never removes it. Real protection is limiting what a fooled model can do.
Myth
Hiding the API route URL keeps it safe.
Fact
Anyone can see network requests in the browser's developer tools. The server must check every request as if it came from an attacker.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the threat model as a living table in the repository (door, worst case, control, test) and update it in the same pull request that adds a tool, a data source or a new place text is shown. The review question becomes "which row changed?".
- ▸
Score each risk by the worst case through the door, not by how likely the attack looks. A tool that can issue refunds deserves strict controls even if nobody has ever attacked you, because the cost of being wrong is the refund.
- ▸
Security work on AI features is mostly ordinary web security applied to a new kind of input: validation, authorisation, output encoding, logging. The new part is only that the "user" now includes text the model repeats from other sources.
Remember this
- 1
The problem this solves: Teams add an AI feature, test it with friendly questions and ship. Then someone pastes a web page that contains hidden instructions, or a customer message tries to make the assistant reveal other people's orders. Nobody asked "who can put text in front of the model, and what could it make our app do?" A threat model is the habit of asking that before shipping. This topic gives you the map the rest of the phase follows.
- 2
A threat model is a short written answer to four questions. What do we protect (data, money, accounts, reputation)? Who can reach it and how (the doors)? What is the worst thing that could happen through each door? What stops it, and how do we check that it works? It does not need special tools. A table with those four columns is a threat model.
- 3
The doors. The model reads one long piece of text called the prompt. Parts of it come from you: the system prompt (your standing instructions). Everything else comes from outside: the customer's message, a document they attach, a web page Mitra fetched, a tool result, an earlier message in the conversation, a row from your database that another customer wrote. An attacker needs control over only one of these.
- 4
Trusted and untrusted. Text you wrote and shipped is trusted. Everything else is untrusted: you cannot predict it. Untrusted does not mean evil. A customer pasting an email is not attacking you. It means your code must be safe whatever the text says, because the day it is hostile you will not get a warning.
- 5
Why the model cannot protect you. To a language model, your system prompt and a pasted document are both just text in the same prompt. It predicts a sensible continuation. If the document says "ignore previous instructions", that may be a sensible continuation too. Telling the model "never obey the document" lowers the chance but does not make it zero, and you cannot test every phrasing. So the design rule is: assume the model will sometimes be fooled, and make sure that being fooled can only do small, visible harm.
- 6
What attackers want. Four goals cover most cases. (1) Data: other customers' orders, your system prompt, the user's private details. (2) Actions: cancel an order, send an email, spend money through your tools. (3) Cost: make your feature burn tokens (budgets are Phase 13). (4) Trust: get your assistant to say something false or offensive under your brand. Each later topic in this phase blocks one of these.
- 7
The standard checklist. OWASP, the security community's reference body, publishes the OWASP Top 10 for LLM Applications (the 2025 edition is current when this was written). The entries this phase touches are LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage and LLM10 Unbounded Consumption. Use it as a checklist to compare your own table against, not as a substitute for thinking about your app.
- 8
The browser is untrusted too. Everything the browser sends can be forged. A request that claims
role: "system", auserIdfield, anapproved: trueflag or a 300,000-character message is simply data until your server checks it. Security checks that exist only in the React code protect nobody who calls your route directly.
Explain it without notes
What is a threat model and what four questions does it answer?
Why can't an AI feature rely on the model to separate your instructions from untrusted text?
Name the untrusted inputs of a shop assistant that can read orders and summarise pasted documents.
Why must a security check live on the server and not only in the React code?
Practice
Make a threat-model table for Mitra as it is today (answers questions about orders). Give four rows with columns door, worst case, control and test.
Extend the second example so a request with more than 20 messages is refused with 413 and the message conversation too long. What does that limit protect?
Trade-offs
- ↔
More server-side checks mean more code and a little latency, but each one is a cheap boolean test compared with a model call. The cost of skipping them shows up as incidents.
- ↔
A strict input limit protects cost and abuse but may reject a legitimate long message. Pick the limit from real usage and show a clear message (Topic 13.2) instead of failing silently.
Done when you can
I can list every source of text that reaches the model in a feature and mark who controls it.
I can explain why the model cannot tell instructions from data.
I can name the four things an attacker wants from an AI feature.
I can write a four-column threat-model table for a feature.
I can explain why every check must be repeated on the server.