Topic 11.2
Logging Prompts and Responses Safely
In one line
Prompts and replies are the most useful thing to log and the most dangerous: they contain names, emails, secrets and private documents. Safe logging uses structured lines, a redactor before the write, sampling, and a retention limit.
Think of it like this
A shop keeps a notebook of every customer complaint so it can fix problems. If the notebook also records each customer's phone number, card number and home address, a lost notebook becomes a disaster. The sensible shop writes the complaint and a reference number, hides the private details, and throws old pages away after a month.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Structured log
- A log line with fixed named fields (usually one JSON object per line) that a program can filter.
- Redaction
- Replacing private or secret values in text with a placeholder such as [EMAIL].
- PII
- Personally identifiable information: names, emails, phone numbers and other data that points to a person.
- Sampling
- Keeping only a random fraction of events (here, the full text of some healthy calls).
- Retention
- How long data is kept before it is deleted automatically.
- Request id
- A unique id attached to every record of one request, so its records can be joined.
- Seeded generator
- A random-number routine that starts from a fixed number, so it gives the same sequence every run.
Step by step
01Step 1: the naive log line
The simplest log writes the prompt as it arrives. The first line in the example does that, and the output shows an email address, a phone number and an API key sitting in the log in plain text.
02Step 2: write one JSON line per event
log_line builds one object with fixed keys first (ts, level, event) and then the event's own fields, and prints it as compact JSON. Same key order every time means lines are easy to compare and search.
line = {"ts": clock_ms, "level": level, "event": event}
line.update(fields)
print(json.dumps(line, separators=(",", ":")))03Step 3: redact before the write
redact replaces each match of three patterns with a placeholder. The second printed line is the same call with a redacted prompt, plus the metadata an engineer needs: model, token counts, latency and status. The private values are gone, and nothing is lost that debugging needed.
04Step 4: sample what you keep
The second example decides per call whether to keep the full text. Errors and timeouts are always kept (that is where you need the text). Healthy calls are kept when a seeded random draw falls below the sample rate. The same seed gives the same decisions in every language.
05Step 5: apply retention
Each stored log has an age in days. The retention rule deletes full-text logs older than 30 days. In a real system this is a scheduled job or a storage lifecycle rule, not code you run by hand, but the rule itself is this simple.
06Step 6: know what the patterns miss
Patterns catch formats. They do not catch a name in a sentence, a street address, or "my diagnosis is...". For sensitive products, either log no text at all, or run a stronger detector first (Topic 10.5). Redaction reduces risk; it does not remove it.
Try it yourself
- 1
Add a pattern
Add a redaction for a 16-digit card number (
\b\d{16}\bbecomes[CARD]). Add a card number to the prompt and check that the redacted line hides it. Which order of the three replacements could go wrong if a pattern matched part of another? - 2
Change the sample rate
Set
SAMPLE_RATEto 0.5 and predict which requests are kept from the draws printed earlier (0.001, 0.571, 0.448, ...). Then run. - 3
Try a different seed
Change the starting
statefrom 42 to 7. How many of the 8 calls are kept now? Is any seed better than another for sampling in production, or only different?
Code & diagrams
The first line logs the prompt as it arrived. The second redacts it and adds the metadata. In Java the course's Json helper writes the compact JSON (real projects use Jackson).
Expected output
{"ts":1000,"level":"info","event":"llm_call","request_id":"r1","prompt":"Hi, I am ana@example.com (call 9876543210). My key is sk-test12345678. Summarise my invoice."}
{"ts":1000,"level":"info","event":"llm_call","request_id":"r1","model":"small","tokens_in":38,"tokens_out":61,"latency_ms":640,"status":"ok","prompt":"Hi, I am [EMAIL] (call [PHONE]). My key is [KEY]. Summarise my invoice."}Seeded generator: state = state * 48271 % 2147483647. Errors and timeouts are always kept, healthy calls are kept 1 time in 4.
Expected output
r1 ok: keep (draw 0.001)
r2 ok: drop (draw 0.571)
r3 error: keep (error: always keep)
r4 ok: drop (draw 0.448)
r5 ok: drop (draw 0.654)
r6 timeout: keep (error: always keep)
r7 ok: drop (draw 0.629)
r8 ok: drop (draw 0.716)
kept 3 of 8 full-text logs
deleted by the retention rule: r6Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to keep the errors
In should_keep, delete the if status != "ok": return True, ... lines, so errors are sampled like everything else.
Myth vs fact
Myth
Redaction makes logs safe.
Fact
It handles known formats only. Names, addresses and free-form details slip through, so combine redaction with sampling, short retention, and no text at all for sensitive features.
Myth
Debug logs in production are harmless if nobody reads them.
Fact
They are copied to backups, search tools and tickets, and they leak the first time someone exports them. Treat every log as data that can leave.
Myth
Sampling hides bugs.
Fact
Sampling healthy calls does not; sampling errors would. Keep every failure and metadata for every call.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Redact at the edge of the logging code, not at each call site. A single logging function that redacts means a new feature cannot forget to do it. Add a test that feeds it a prompt full of fake emails and keys and checks none survive.
- ▸
Hash user ids with a keyed hash if you need to correlate one user's calls without storing who they are. Keep the key out of the logs.
- ▸
Keep a per-request opt-in to full text: when a user reports a problem and agrees, turn on full logging for that request id for a day. It gives debugging power without logging everything.
Remember this
- 1
Why this exists. You need the prompt and the reply to debug a bad answer (Topic 11.1). But a prompt can hold anything a user typed or a document you retrieved: emails, phone numbers, health details, even API keys pasted by mistake. Logs are copied to many places (search tools, backups, laptops, support tickets). More people read them than read the database. Anything private that reaches a log is now in all those places.
- 2
Structured logs. Write each event as one line of JSON (the key-value text format from Topic 4.2) with the same keys every time:
{"ts":1000,"level":"info","event":"llm_call","request_id":"r1",...}. A machine can then filter by key ("all errors for model small") and a person can still read it. Free-text lines like"call took long"cannot be searched reliably. Real Java projects use a logging library with a JSON encoder; the course'sJsonhelper keeps the example runnable in the browser. - 3
Redact before writing. A redactor is a function that masks private values in the text before it goes into the log. Emails become
[EMAIL], 10-digit phone numbers become[PHONE], and strings that look like keys (sk-...) become[KEY]. It must run before the write, because you cannot un-write a log line. A pattern list catches the obvious cases; it does not catch everything (names, addresses, free-form health text), which is why the next two rules also matter. Topic 10.5 covers detecting personal data in more depth. - 4
Sample the healthy calls, keep the failures. Logging the full text of every call is expensive and increases risk. A common rule: keep the full prompt and reply for every error and for a small fraction (say 1 in 20) of successful calls. For all calls, keep only the small metadata fields (tokens, latency, status). The sample must be random, not "the first N", or it shows only a biased slice. The example uses the course's seeded generator so the result repeats.
- 5
Retention and access. Decide how long full-text logs live (for example 30 days) and delete them automatically; the metadata can live longer. Limit who can read them. Tell users in your privacy policy what you keep. If a user asks you to delete their data, you need to be able to find it, which is another reason to log a user id as a hashed or opaque value, not an email.
- 6
What never goes into a log: API keys and passwords, whole documents from private stores (log ids and chunk numbers instead), and anything your provider or your legal rules forbid. The safest log line for a sensitive feature has no text at all, only sizes, ids and outcomes. This site's own AI tutor works this way for usage: it keeps a count of calls per user per day, not the questions.
Explain it without notes
Why must redaction happen before the log line is written, not when someone reads the log?
Why do you always keep failed calls but only sample healthy ones?
What are the limits of pattern-based redaction, and what do you do about them?
Practice
Write mask_email(text) that replaces each email with its first letter, *** and the domain, so ana@example.com becomes a***@example.com. Use re.sub with a function.
Write is_expired(age_days, kept_days) and purge(logs, kept_days) where logs is a list of (request_id, age_days); purge returns the ids of logs to delete (older than kept_days).
Trade-offs
- ↔
Full-text logs make debugging easy and raise privacy risk; metadata-only logs are safe and make some bugs hard to find.
- ↔
A higher sample rate catches more rare problems and stores more private text.
- ↔
Long retention helps investigate slow-burning issues and increases what a leak exposes.
Done when you can
Done when you can write a log event as one line of JSON with fixed keys.
Done when you can write a redactor and say what it does not catch.
Done when you can sample healthy calls and always keep failures.
Done when you can state a retention rule and what must never be logged.