Topic 10.5
Personal Data in Documents
In one line
Documents contain names, emails, phone numbers and card numbers that nobody meant to put in a search index. Find and replace them before chunking, keep them out of embeddings and logs, and decide how long anything is kept.
Think of it like this
Before a teacher photocopies a class's worksheets for another school, she covers each pupil's name with a strip of tape. She does it before copying, because a name that reaches the copy cannot be un-copied. Redaction in RAG works the same way: it happens before the text is chunked, embedded, cached or logged.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- PII
- Personally identifiable information: data that can identify a person, such as a name, email address, phone number or card number.
- Redaction
- Replacing sensitive values in text with a label, for example [EMAIL].
- Pseudonymisation
- Replacing a value with a stable token so records can be linked without revealing the person.
- Recall (of a redactor)
- The share of the real personal data it found. Low recall means leaks.
- Precision (of a redactor)
- The share of its redactions that were really personal data. Low precision ruins useful text.
- Retention
- How long data is kept before it is deleted.
- Data minimisation
- Collecting and keeping only what you need.
Step by step
01Step 1: find where personal data can come from
Make a list of your sources and mark those likely to hold personal data: tickets, chat logs, forms, uploaded files, email exports. The handbook has none; tickets do.
02Step 2: write patterns for values with a shape
Each pattern is a label and a regular expression. Keep them simple and test each on a few real-looking fake values. Order matters: a card number is found before a phone number, so digits are not claimed by the wrong pattern.
03Step 3: add a list for names you know
The customer table gives exact names. Replace them with [NAME]. This misses a name you do not hold, which is why free text also needs a model or a human sample.
04Step 4: redact before chunking
Call redact on the whole document in the ingestion pipeline. Count what was found per type. A sudden change in the counts (zero emails in a ticket batch) is a sign that a format changed and the patterns no longer match.
05Step 5: look at what slipped through
The third ticket still contains an address written in words. Add it to a review sample, widen the rules if the pattern is common, and accept that some will always be missed.
06Step 6: set retention and keep the mapping apart
Decide how long raw tickets, redacted chunks and logs are kept. If you pseudonymise, store the token table separately, with stricter access than the index, and delete it when the person asks to be forgotten.
Try it yourself
- 1
Add a postcode
Add a pattern for a made-up postcode form such as
AB1 2CDwith letters and digits you choose, and a ticket that contains one. Does it also redact something that is not a postcode? Add a ticket that mentions a bike code with the same shape, such asBK7 4XY, and see what happens to it. - 2
Break the order
Move the
PHONEpattern aboveCARD. Do any card numbers survive or become part of a phone match? Print the result for ticket 2. - 3
Make a new miss
Write a ticket where the address is split over two lines or uses spaces around the at sign. Does the redactor catch it? What would you change, and what would that cost in false matches?
Code & diagrams
All values are invented. Real redactors also handle other phone and card formats, and check a card number's checksum to avoid redacting ordinary numbers.
Expected output
ticket 1: [NAME] wrote from [EMAIL]: the lock on bike 12 is stuck, call me on [PHONE].
ticket 2: Charged twice on card [CARD] for the monthly pass. [NAME], [EMAIL]
ticket 3: My mail is jane dot doe at example dot com, please reply there.
found: CARD=1 PHONE=1 EMAIL=2 NAME=2Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A pattern that is too narrow
In the Python tab, change the email pattern to [A-Za-z0-9]+@[A-Za-z0-9-]+[.][A-Za-z]{2,} (no dot, underscore or hyphen allowed before the at sign). Run.
Myth vs fact
Myth
Embeddings are just numbers, so they are anonymous.
Fact
A vector is derived from the text and can sometimes be partly reversed. Give vectors the same protection, retention and deletion as the text they came from.
Myth
We redact the answer, so the index can hold anything.
Fact
The index, cache, prompts and logs would all contain the raw data, and a deletion request would have to reach every one. Redact at ingestion.
Myth
A good redactor finds everything.
Fact
Every redactor has misses and false matches. Measure recall and precision on real samples and plan for the leftovers.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure the redactor like a retrieval system: build a labelled sample of real (access-controlled) texts, count found, missed and wrongly redacted values, and set a floor in CI. Re-measure after each pattern change. Recall matters more than precision for security; precision matters for usefulness.
- ▸
Card numbers should not be in your documents at all. If you handle payments, the payment provider's tokenised flows keep the number away from your systems. Redaction is the safety net, not the plan. Check obligations such as PCI DSS with your compliance owner; this course gives no legal advice.
- ▸
Laws such as the GDPR give people rights to access and erase their data and require you to limit what you keep. The engineering consequence is the same everywhere: know where personal data lives (a data map), keep it in few places, and have a tested way to delete it (Topic 10.6).
- ▸
Be careful with hashes of personal values as ids: an unsalted hash of an email can be reversed by trying known addresses. Use a keyed hash (HMAC) with a secret, stored outside the index, when you need stable pseudonyms.
Remember this
- 1
Why it is needed. Support tickets, chat exports and forms are among the most useful sources for a RAG system, and they are full of personal data (also called PII, personally identifiable information): names, emails, phone numbers, addresses, card numbers. Once it is in the index it spreads: into chunks, vectors, caches, prompts sent to a model provider, logs and backups. Each copy is a place it can leak from and a place you must clean when a person asks to be deleted (Topic 10.6). LLM Engineering Topic 10.5 covers redaction for any LLM app; this topic is about what retrieval adds.
- 2
Redact at ingestion, the earliest point. The cheapest copy to control is the first one. Put a redactor in the ingestion pipeline between loading and chunking (Topic 4.9), so everything downstream (chunks, embeddings, search results, prompts, logs) only ever sees the cleaned text. A later redaction (on the answer, in the log) is still worth having, as a second layer.
- 3
Patterns for things with a shape. Emails, phone numbers and card numbers have a recognisable form, so regular expressions (patterns written in a small language for matching text; Python's
remodule runs them) find them. The example uses the clearly fakejane.doe@example.com,+1 555 010 4477and4111 1111 1111 1111(a well-known test card number). It replaces each with a label such as[EMAIL]. The label keeps the sentence readable for search and for the model, and loses the value. - 4
Names and addresses need a list or a model. No pattern recognises a name: "Jane Doe" looks like any two capitalised words. The example uses a list of known customer names from the customer table, which is accurate and cheap for names you already hold. For free text about people you do not hold, production systems use named-entity recognition models. These are measured on recall (how much PII they find) and precision (how often they wrongly redact an ordinary word), and neither is 100%.
- 5
What the patterns miss. The third ticket writes the address in words ("jane dot doe at example dot com") and passes through untouched. Any redactor has misses: odd spellings, digits split by newlines, other countries' formats. So treat redaction as risk reduction, not a guarantee. Sample real redacted output regularly and have a person look, add a second check that scans the final index and logs for patterns, and keep the highest-risk fields (card numbers, passwords) out of the source in the first place.
- 6
Keep personal data out of vectors and logs. A vector is a numeric summary of a text, and research has shown that embeddings can sometimes be partly inverted to recover words, so do not assume a vector is anonymous. Apply the same retention and deletion rules to vectors as to text. For logs (Topic 11.6), log ids and scores rather than the full question when possible, redact the rest, and set a retention period: delete logs after the number of days you need them for debugging, no longer.
- 7
Pseudonyms when you still need to tell people apart. Sometimes the system must know "the same person wrote twice" without knowing who. Replace each value by a stable token (
[EMAIL_1]) and keep the table that maps tokens back to values in a separate, restricted store, never in the index. That table is itself personal data, and it is what a deletion request removes.
Explain it without notes
Why should personal data be removed before chunking and embedding, not from the final answer?
How do you find an email address, and how do you find a person's name? Why are they different?
Name two ways a redactor can fail, and what you do about each.
What is pseudonymisation, and where must the mapping table live?
Practice
Write mask_emails(text) that replaces every email address of the form letters, digits, dots, underscores and hyphens, an at sign, and a domain with [EMAIL], and returns the new text. Test it on a sentence with two addresses.
Write pseudonymise(texts) that replaces each distinct email with [EMAIL_1], [EMAIL_2], ... in order of first appearance and returns the new texts and the mapping dict.
Trade-offs
- ↔
Aggressive redaction: fewer leaks and some useful text lost. Tune with measured recall and precision.
- ↔
Labels like [EMAIL]: readable and searchable, but identical for everyone. Pseudonyms keep people distinguishable, at the cost of a sensitive mapping table.
- ↔
Long log retention: easier debugging, larger exposure. Keep only what you need and for as short a time as you can.
Done when you can
Done when you can place a redactor in an ingestion pipeline before chunking.
Done when you can write patterns for emails, phones and cards and test them on fake data.
Done when you can say what patterns miss and how you would measure and cover it.
Done when you can explain why vectors and logs need the same care as text and state a retention rule.