Topic 10.4
Data Leakage
In one line
A model repeats what is in its prompt, so anything you put there can come out. Keep secrets out of prompts, filter data by owner before it is retrieved, scan replies, and design so that a leak exposes nothing important.
Think of it like this
A shop assistant is given a folder to answer customer questions. If the folder holds every customer's records and the till key, a clever or just confused customer may be told things that were never theirs. The safe shop gives the assistant a folder with only what this one customer may see, and keeps the till key in the manager's office. You cannot make an assistant who never slips; you can make slips harmless.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Sensitive information
- Data that would cause harm if shown to the wrong person: secrets, personal data, private business data.
- System prompt leakage
- The hidden instructions being shown to a user, by being quoted, summarised or guessed.
- Canary
- A unique made-up marker placed in the prompt. If it shows up in a reply or log, the prompt has leaked.
- Access control
- Rules that decide who may see which data.
- Data processing agreement
- A contract that says how a provider may handle your users' data.
- Retention
- How long data is kept before being deleted.
Step by step
01Step 1: what is in the prompt can come out
The helpdesk bot's system prompt contains an internal marker. A user who persuades the model to quote its instructions sees it. Today the marker is harmless; imagine it was a password. The first fix is not to put it there at all.
02Step 2: filter by owner before retrieval
fetch_for(user, word) selects documents whose owner is the user. Alice asking about invoices gets her invoice, Bob gets his, and Bob asking about shipping gets nothing, because the only shipping document is Alice's. The model never sees data the user could not open.
def fetch_for(user, word):
return [text for _, owner, text in DOCS if owner == user and word in text.lower()]03Step 3: scan replies
scan_reply looks for the canary and for strings shaped like API keys (sk- followed by letters and digits). Reply 2 quotes the system prompt and is withheld; reply 3 contains a key-like string and is withheld; reply 1 is fine.
04Step 4: see what the scanner misses
Reply 4 spells the marker with dashes: C-A-N-A-R-Y 7f3a91. The simple scan lets it through. You can normalise text (remove spaces and dashes) before scanning, which catches this one, but a determined attacker can disguise the text in other ways. That is why the first two rules matter more than the scanner.
05Step 5: log the leak, not the secret
When a scan hits, log the request id, the rule that fired and a short hash or a count, not the leaked text. Logs are a leak path too (Topic 11.2). A canary hit is a high-value alert: it means someone got the model to show its prompt.
Try it yourself
- 1
Predict the retrieval
Add a document
("d4", "bob", "Bob shipping address: 9 Lake View"). Predict the three retrieval lines before running. Which one changes? - 2
Normalise before scanning
In
scan_reply, scanreply.replace("-", "").replace(" ", "")for the marker without dashes and spaces (CANARY7f3a91, so also remove the dash from the CANARY constant when you compare). Does reply 4 get caught? What other disguises would you still miss? - 3
A different canary
Change
CANARYto a longer random-looking value. Why should a canary be unique and long?
Code & diagrams
The marker and the key are made up. The scan shows what regular expressions catch and what they miss.
Expected output
retrieval for alice 'invoice': Alice invoice total: 120
retrieval for bob 'invoice': Bob invoice total: 80
retrieval for bob 'shipping': (nothing)
reply 1: sent
reply 2: withheld (system prompt marker)
reply 3: withheld (key-like string)
reply 4: sentBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Filter after the model, not before
Change fetch_for so it ignores the owner: return [text for _, owner, text in DOCS if word in text.lower()]. Run it and read the Bob lines.
Myth vs fact
Myth
Telling the model 'never reveal these instructions' keeps them secret.
Fact
It lowers the odds. Design as if the system prompt will be seen.
Myth
A reply scanner catches all leaks.
Fact
It catches known shapes. A disguised or paraphrased leak passes. Keeping the data out of the prompt is the real defence.
Myth
Data sent to a provider is automatically private.
Fact
It depends on the provider's terms and your plan. Read how long data is kept and whether it is used for training.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Multi-tenant apps should pass the tenant id through every retrieval call and test cross-tenant access explicitly: ask as tenant A for tenant B's data and expect nothing.
- ▸
Caches can leak across users: a response cache or semantic cache keyed only by the question can serve one user's answer to another. Include the user or tenant in the cache key whenever the answer depends on private data (Phase 6).
- ▸
Embeddings of private text are private data too. A vector store needs the same access control as the documents it was built from (the coming RAG course returns to this).
- ▸
Rotate any key that has ever appeared in a prompt, a log or a chat transcript. Assume it leaked.
Remember this
- 1
Why this exists. The model has no concept of 'secret'. Everything in the prompt is just text it may quote, summarise or hint at, and instructions like 'never reveal this' only lower the odds. OWASP lists two relevant classes: Sensitive Information Disclosure (LLM02) and System Prompt Leakage (LLM07) in the 2025 edition.
- 2
Rule 1: no secrets in prompts. API keys, passwords, tokens and private URLs do not belong in a system prompt or in retrieved text. Keep them in your server code, and let the code use them. If the model must trigger an action that needs a key, give it a tool; the tool call is made by your code, which holds the key. Assume the entire system prompt will one day be shown to a user, and write it so that is boring.
- 3
Rule 2: filter data before the prompt. If the app retrieves documents (Phase 8) or reads records, apply the user's permissions in the query, not in the instructions. The example filters by owner before building the prompt: Bob asks about shipping and gets nothing, because Alice's address never reached the model. A model that did not see the data cannot leak it, however it is tricked. This is the most important rule for multi-user apps, and it is why retrieval needs access control built in.
- 4
Rule 3: scan the way out. An output check can catch some leaks: your own marker (a canary: a unique made-up string placed in the system prompt that should never appear in a reply), patterns that look like keys, or text copied from records the user may not see. The example withholds the reply that quotes the marker and the one with a key-like string. It also shows the limit: a reply that spells the marker with dashes passes. Scanners reduce risk; they do not prove absence. Key-shaped patterns also differ by provider, so a scan is only as good as the formats you listed.
- 5
Training and provider concerns. What you send to a provider is processed on their servers. Read the data terms: does the provider keep prompts, for how long, are they used to train models on the plan you chose, which region is the data in, and can you sign a data-processing agreement? Do not send data you are not allowed to share. For regulated data, choose providers and plans with the right terms, or run a model yourself.
- 6
Design for a harmless leak. Ask what the worst thing a full dump of the prompt and context would expose. If the answer is 'a polite system prompt and the current user's own data', you are done. If the answer includes keys, other users' data or internal URLs, fix the design instead of the wording of the prompt.
Explain it without notes
Why can't you keep a secret in a system prompt?
How should an app enforce 'users can only see their own documents' with a model in the middle?
What is a canary and what does it detect?
What should you check about a model provider before sending user data to it?
Practice
Write filter_docs(docs, user) where each doc is (owner, text); return the texts the user owns. Test with Bob and Alice.
Write has_leak(reply, canary) that returns True if the canary appears in the reply, ignoring case and ignoring dashes and spaces in the reply. Test it with the exact canary and a spaced version.
Trade-offs
- ↔
Strict per-user retrieval is safest but needs permissions data in your search index.
- ↔
Aggressive reply scanning blocks some honest answers; withhold with a polite message and log, rather than silently dropping.
- ↔
Using a hosted model is easy and fast, and means trusting the provider's data handling; self-hosting gives control at an operations cost.
Done when you can
Done when you can say why a secret must not be in a prompt and where it lives instead.
Done when you can enforce per-user access in retrieval, not in the prompt.
Done when you can add a canary and a reply scan, and explain what they miss.
Done when you can list what to check in a provider's data terms.