PHASE 9Advanced ~27 min· topic 8 of 8

Topic 9.8

Build: Mitra Under Attack

In one line

Mitra gets its defences: a server that builds the system message and the tools for the signed-in user, a screen that renders replies as text with checked links and no outside images, and a short checklist you run to prove each defence works.

Think of it like this

A shop that has been broken into once. The owner does not just hope. She puts a lock on the stockroom, keeps the cash in a safe, labels what is for sale and what is not, and then walks around each evening ticking a list to check every lock. Mitra now gets its locks, and the list you tick to prove they hold.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Hardening: Making a system harder to misuse by closing doors and limiting powers.
Audit: A check that looks at the real result (the DOM, the log, the tool output) and reports facts.
Regression test: A test that keeps a fixed bug fixed. A hostile sample plus an assertion is one.
Hostile sample: A harmless piece of text that imitates an attack, kept in the test suite.
Defence in depth: Several independent protections, so one failing does not break everything.
Scripted model: The stand-in used in this course. It plays a fooled model so you can test your defences.

Step by step

01Start from Mitra with tools

Mitra has useChat on the client and streamText on the server, and a getOrder tool. The route currently trusts nothing it should not, but nothing proves it. We add the hardening and then the proof.

02Harden the route

The route reads the session user from the server (here a constant, in a real app from your auth library). It builds toolsFor(session), passes a system message that names <document> as data and caps steps with stopWhen: stepCountIs(3). It counts suspicious lines in the user's text and logs the count.

route (the changes)whole filejs

const session = "asha"; // from the signed-in session, never from the request body
const result = streamText({
  model,
  instructions: SYSTEM,                 // built here, not by the browser
  messages: await convertToModelMessages(messages),
  tools: toolsFor(session),             // this user's tools only
  stopWhen: stepCountIs(3),             // a fooled model cannot loop forever
});

03Harden the screen

Every text part goes through renderText. Links go through safeHref and images become notes. Tool parts show a sentence, not JSON. The container has the id #messages, so the audit can inspect it.

04Run the attack

The first example sends a message that contains a hostile instruction. The route logs 1 suspicious line. The scripted model asks for ORD-2001 (another customer's), the tool answers found: false, and the hostile reply arrives. When the stream ends, the audit prints what is really in the page.

Run the attackdiagram

Rendering diagram…

05Run the checklist

The second example runs ten small checks, one per checklist item, with a hostile sample for each. A check that fails prints FAIL with the reason. Put the same kind of checks in your project's test suite and run them on every change.

Try it yourself

1
Change who the server thinks is signed in
In the first example, change const session = "asha"; to const session = "ravi"; and run it.
The tool part now returns Ravi's item: whoever the server believes is signed in decides the data, which is why that value must come from real authentication.
2
Allow every protocol
In the first example, change the safeHref list to include "javascript:" and run it.
The audit now lists a second href, so the link got through your helper. Its text is not the original: React 19 replaces any javascript: address with a script that only throws an error. That is a last line of defence in one library, not a reason to loosen your own list. One entry in an allow-list reopens the door.

Code & diagrams

Mitra under attack

Everything hostile here is a harmless stand-in: `alert(1)`, a made-up address and a scripted model. The `(server)` style logging is the route's own console. The audit reads the real DOM.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

server: suspicious lines in the user's text: 1
audit: scripts 0, images 0, hrefs ["https://chai.example/track/1042"]
tool part: {"found":false}
script text visible: true

Mitra's safety checklist

These stand-ins are tiny copies of the real helpers. In your project the same checks import the real functions and run in your test runner on every change.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

PASS identity comes from the session
PASS system message is built on the server
PASS forged system role is dropped
PASS missing and foreign orders look the same
PASS javascript: link is refused
PASS https link is kept
PASS outside image is refused
PASS own image is allowed
PASS log line has no words
PASS filter stop has its own message
all 10 checks pass

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the protocol allow-list

In the first example, replace the line return ["https:", "mailto:"].includes(u.protocol) ? u.href : null; with return u.href; and run it.

terminal

$ Console, the audit line

── what you'll see ──

audit: scripts 0, images 0, hrefs ["https://chai.example/track/1042","javascript:throw new Error('React has blocked a javascript: URL as a security precaution.')"]

Myth vs fact

Myth

Once the defences are written, the feature is secure.

Fact

Secure means tested and watched. New tools, new data sources and new rendering code each reopen questions. Keep the checklist in the suite.

Myth

A fooled model in the test proves the real model is fooled the same way.

Fact

The scripted model is the worst case on purpose. If your code survives it, it survives gentler models; the test does not tell you how often a real model is fooled.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Turn the checklist into a pull-request template. Anyone adding a tool, a data source or a renderer ticks which row they touched and links the test. Reviews then ask about security because the form asks.
▸
Add a monthly review of real traffic, within your privacy rules: counts of suspicious-line hits, denied lookups, blocked images and filter stops. A change in any of them is the earliest sign of a new attack or a broken defence.
▸
Keep a small red-team set of hostile samples as data files (a document with hidden instructions, a reply with odd links, a message with invisible characters) and run them against the real model on a schedule. Assert on outcomes, not wording, and treat a new failure as a bug in your system, not in the model.

Remember this

1
The problem this solves: Phases 0 to 8 made Mitra useful. A useful assistant that reads documents, calls tools and shows Markdown is also a target. This topic puts the defences from Topics 9.1 to 9.7 in one place and, more importantly, adds the tests that show they work. A defence you cannot show working is only a hope.
2
Starting point. Mitra from Phase 8 streams answers with useChat and streamText, renders formatted answers (Phase 6) and can look up an order and ask approval before cancelling (Phase 8). Its route is /api/chat and its screen has #messages, #status and #send. We assume that code and add the hardening around it.
3
What the route now does. (1) It takes the identity from the session and never from the request body. (2) It builds the system message itself, with the rule that text inside <document> is data. (3) It scans the user's text for instruction-like lines and logs how many (a count, not the text). (4) It builds the tools for this user only, so getOrder can see only that customer's orders. (5) It limits the number of steps.
4
What the screen now does. It renders every text part through one function that turns Markdown into React elements. A link survives only if its protocol is https: or mailto:. An image is replaced by a visible note, because Mitra has no reason to show model-chosen images. HTML in a reply appears as characters. A tool part shows a human sentence, and a failed lookup shows "not found", the same for missing and foreign orders.
5
The live example is a real attack, stopped. The scripted model is fooled by a hostile document: it asks for another customer's order, then writes a reply that contains a script tag, a javascript: link, and an image pointing at attacker.example with an order number in the address. All three are stand-ins, nothing real. The audit afterwards counts scripts, images and links in the real DOM, and prints what the tool returned.
6
What Mitra can do now that it could not before: read untrusted text without giving it power, refuse to show other customers' orders even when fooled, show model output without running or loading anything it did not choose, and prove each of those with an audit. What is still missing and which phase adds it: a polished interface (Phase 10), file and image input (Phase 11), on-device models (Phase 12), usage limits and budgets (Phase 13), and long-running agents (Phase 14).
7
Your checklist, in order. (1) Identity comes from the session. (2) The system message is built on the server. (3) Roles other than user and assistant are dropped. (4) Tools are built per user and return the same answer for missing and foreign records. (5) Replies render as text. (6) Links pass a protocol allow-list. (7) Model-supplied images are not shown. (8) Logs hold ids and sizes, not words. (9) A refusal or filter stop shows its own message. (10) Each of these has a test with a hostile sample. The second example runs ten such checks on small stand-ins.

Explain it without notes

List the defences added to Mitra in this phase and the attack each one stops.

Why does the audit read the DOM instead of the code?

What would you test if you added a cancelOrder tool next?

Practice

Add an eleventh check to the checklist example: a message longer than 2000 characters is refused. Write the check and a tiny acceptMessage function.

Add a check that an image whose address uses http: (not https:) from your own host is refused. Does the current imageAllowed pass it?

Trade-offs

↔
Showing no model-chosen images makes answers plainer, and removes a whole class of leaks. If you need pictures, send product images from your own data, not from the reply text.
↔
A full checklist in the test suite slows each change by a little. A security incident costs far more than the time it takes to run ten checks.

Done when you can

I can describe how the route gets identity, the system message and the tools.
I can describe how the screen renders text, links, images and tool results safely.
I can run an audit of the real DOM after a hostile reply.
I can write a checklist with a hostile sample and an assertion for each defence.
I can say what Mitra still lacks and which phase adds it.

Command Palette