PHASE 10Advanced ~20 min· topic 3 of 9

Topic 10.3

Treating Output as Untrusted

In one line

A model's reply is text produced from inputs you do not control, so it is as untrusted as a form field. Escape it before putting it in a web page, pass it to databases as parameters, and let it select from an allow-list instead of building commands.

Think of it like this

A school office takes a message from a visitor and pins it on the public noticeboard. A careful office reads it as a message. A careless one lets the visitor write the board's own headings and arrows, so the visitor can redirect the whole school to the wrong room. The fix is not to trust the visitor more; it is to put every message in a frame that makes it a message, whatever it says.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Escaping: Replacing special characters with safe codes so text is shown as text, not read as markup or code.
HTML entity: A code such as < that a browser shows as the character < without treating it as a tag.
XSS (cross-site scripting): Page text that a browser runs as script because the site inserted it unescaped.
Parameterised query: A database call where the SQL text and the values are sent separately, so values can never become SQL.
Allow-list: A fixed list of permitted values. Anything else is refused.
Improper output handling: Using a model's reply in another system without validating or encoding it.
Content Security Policy: A header that tells the browser which scripts and sources a page may use.

Step by step

01Step 1: see the problem

Suppose a document made the model write <script>show('banana')</script>, and the app puts the reply inside <p>...</p>. The first output line shows the page text: to a browser, that is a script tag, not a sentence. The word show is a made-up function, so nothing real runs, but with a real function it would.

02Step 2: escape before inserting

escape_html replaces the five special characters. The order matters: & first, otherwise the & in < is turned into &lt; and the page shows garbage. After escaping, the page shows the characters literally, which is what you wanted for plain text.

main.pywhole filepython

def escape_html(text):
    return (text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
                .replace('"', "&quot;").replace("'", "&#39;"))

03Step 3: keep values out of SQL text

The string-built query contains the name between quotes, so the apostrophe in O'Brien ends the string early and leaves a broken query. The parameterised version keeps the SQL text fixed and passes the name separately. The program only prints the query and its parameters, because the lesson is the shape of the call; your database driver does the rest.

04Step 4: let the model choose from a menu

The reply has to be exactly ls, pwd or date. rm banana is refused. ls; echo banana is refused too, even though it begins with a legal word, because the match is exact and nothing is ever handed to a shell.

Step 4: let the model choose from a menudiagram

Rendering diagram…

05Step 5: add a second wall

Escaping is a rule every developer must remember every time. Add protections that work even when someone forgets: a Content Security Policy, a database user that can only read what the feature needs, and a sandbox for anything that executes. Defence in depth again.

Try it yourself

1
Predict the escaped line
Add Tom & "Jerry" as a second reply and print both raw and escaped versions. Predict each character that changes before running.
2
Change the menu
Add whoami to SAFE_COMMANDS. Run again. Which line changes? Should a feature run it? Think about what the menu is for.
3
Try a name with two apostrophes
Change name to D'Arcy O'Neil. What does the string-built query look like now, and does the parameterised version change?

Code & diagrams

Escape for HTML, parameters for SQL, a menu for commands New tab

No real database or shell is used: the program prints the page text, the query shape and the decision. The script text is a harmless stand-in.

Expected output

raw page html:     <p><script>show('banana')</script></p>
escaped page html: <p>&lt;script&gt;show(&#39;banana&#39;)&lt;/script&gt;</p>
string-built SQL:  SELECT * FROM users WHERE name = 'O'Brien'
parameterised SQL: SELECT * FROM users WHERE name = ?  params: O'Brien
ls -> would run: ls
rm banana -> refused: rm banana
ls; echo banana -> refused: ls; echo banana

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Escape the ampersand last

In escape_html move .replace("&", "&") to the end of the chain, after the quote replacements. Run it and read the escaped page line.

terminal

$ Run

── what you'll see ──

raw page html: <p><script>show('banana')</script></p>

escaped page html: <p>&lt;script&gt;show(&#39;banana&#39;)&lt;/script&gt;</p>

string-built SQL: SELECT * FROM users WHERE name = 'O'Brien'

parameterised SQL: SELECT * FROM users WHERE name = ? params: O'Brien

ls -> would run: ls

rm banana -> refused: rm banana

ls; echo banana -> refused: ls; echo banana

Myth vs fact

Myth

The reply comes from my own model, so I can trust it.

Fact

The reply is built from the prompt, and the prompt contains text from outsiders. Treat it like any untrusted input.

Myth

I can block bad output by searching for words like script or drop.

Fact

Word lists miss variants and block honest text. Use encoding for the destination (escape, parameters) and allow-lists for choices.

Myth

If the output is valid JSON it is safe.

Fact

Valid shape does not mean safe values. Validate ranges, enums and IDs, and check permissions on the IDs.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Prefer APIs that make the safe path the default: framework text nodes, ORM query builders and prepared statements, process APIs that take an argument list instead of a shell string.
▸
When showing Markdown from a model, sanitise the HTML that comes out, drop raw HTML, and allow links and images only to hosts you list. Remote images and links in rendered output are a known way for injected text to send data out, so decide deliberately what the renderer may fetch.
▸
Add a Content Security Policy that blocks inline script and limits where images and connections may go. It will not replace escaping, but it turns many missed escapes into harmless errors.
▸
Escape for the place the text lands. The five-character HTML escape is right for text between tags and inside quoted attribute values. It is not enough inside a <script> block, inside a style, or in a URL attribute such as href (where a javascript: link needs an allowed-scheme check). Use the framework's binding for that context rather than one escape function everywhere.
▸
Run code a model wrote only in a sandbox with no network, no secrets, a time limit and a memory limit. Treat the sandbox boundary as the security control, not the review of the code.

Remember this

1
Why this exists. Code downstream of the model treats its reply as if you wrote it: you put it in a web page, a database query, a file name or a shell command. But the reply can contain whatever the prompt's untrusted text made it write. OWASP calls the weakness Improper Output Handling (LLM05 in the 2025 list): the model's output reaches another system without validation or encoding.
2
Web pages: escape. If a reply is inserted into HTML as it is, any tags in it become part of the page. Replace the five special characters (&, <, >, ", ') with their entity forms before inserting, or better, use your framework's text APIs (React escapes text children by default; textContent in the browser) and never assign model output to innerHTML. The first part of the example shows <script>show('banana')</script> turned into harmless visible text. Markdown rendering needs the same care: sanitise the HTML a Markdown library produces, and restrict which links and images are allowed.
3
Databases: parameters. If you build SQL by joining strings with model output (or a name the model extracted), special characters in the value change the query. The example uses an honest name, O'Brien: the apostrophe already breaks the string-built query. A hostile value can do far worse, which is why the rule is absolute: never put values into SQL text; send them as parameters (? or $1) so the database always treats them as data. No escaping function you write by hand replaces this.
4
Shells and files: allow-lists. Recall from Topic 7.4: tool arguments are untrusted input, and this is the same rule for any model output. Never pass a model's text to a shell. If the model should choose an action, give it a closed menu (ls, pwd, date) and match the reply exactly against the menu, as the example does. Do not try to detect bad commands by looking for suspicious words; you will miss one. For file names, take only a known identifier and build the path yourself. For URLs, check the host against an allow-list before fetching or rendering.
5
Structured output is still untrusted. A reply that parses as valid JSON with the right fields (Phase 4) is well-formed, not safe. Validate the values too: ranges, lengths, allowed enum values, allowed IDs. And check the user is allowed to act on the ID the model chose.
6
How to verify. Write tests with harmless stand-ins: a reply containing <script>show('banana')</script> must come out escaped in the page; a name with an apostrophe must find the right row; a command not on the menu must be refused. Keep them in the red-team suite (Topic 10.8). Where possible, set a Content Security Policy header on your site as a second wall, so even a missed escape cannot run script.

Explain it without notes

Why is a model's reply untrusted even if you wrote the prompt?

How do you safely show a model's reply in a web page?

Why are parameterised queries better than escaping values yourself?

A feature lets the model pick a command to run. How do you design it safely?

Practice

Write escape_html(text) for the five characters, and print the escaped form of <b>Tom & "Jerry"</b>.

Write choose_action(reply, menu) that returns the reply when it is exactly one of the menu items (compare after stripping spaces and lowering case), and None otherwise. Test with " LS ", "ls -la" and "pwd" against the menu {"ls", "pwd"}.

Trade-offs

↔
Escaping everything is safe but can show ugly entity text if applied twice; encode once, at the last step before the destination.
↔
Allow-lists are strict and safe, but each new capability needs a code change. That friction is the point.
↔
Sandboxes for generated code cost infrastructure and cold-start time, and they are far safer than reviewing the code.

Done when you can

Done when you can escape model text for HTML and explain why & goes first.
Done when you can show a parameterised query and why string-built SQL is unsafe.
Done when you can design an action menu and refuse everything else.
Done when you can explain why valid JSON is not safe JSON.
Done when you can name a second wall behind escaping, such as a Content Security Policy.

Command Palette

Words you'll meet

Step by step

01Step 1: see the problem

02Step 2: escape before inserting

03Step 3: keep values out of SQL text

04Step 4: let the model choose from a menu

05Step 5: add a second wall

Try it yourself

Predict the escaped line

Change the menu

Try a name with two apostrophes

Code & diagrams

Break it on purpose

Escape the ampersand last

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can