“Security found full card numbers in our logs. Twelve teams have read access. How many, since when, and where else?”

1. 10:30

   REPORT

   Scheduled scan flags 16-digit numbers passing the Luhn check in shoplite logs
2. 10:34

   QUERY

   Confirm: find lines containing something that looks like a card number

   A line filter |~ applies a regular expression (RE2 syntax) to the raw line, before any parsing, which makes it fast. \b is a word boundary. In reality you'd also match 13–19 digit ranges and separators.

   LogQL· Lokicopy

   {container="shoplite"} |~ "\\b4[0-9]{15}\\b"

   result

   10:34:01 {"level":30,"msg":"checkout started","card":"4111111111111111","productId":3,...}
   10:34:01 {"level":30,"msg":"checkout started","card":"4111111111111111","productId":1,...}
   ...

3. 10:38

   QUERY

   Size the exposure: how many lines, since when?

   Count matches per hour over the retention window. The first non-zero hour tells you when the bad change shipped, which is the start of the breach window for the incident report.

   LogQL· Lokicopy

   sum(count_over_time({container="shoplite"} |~ "\\b4[0-9]{15}\\b" [1h]))

   result

   {}  21584    (per hour, flat since 2026-09-20 16:00)

4. 10:45

   ACTION

   Contain first: mask at the collector so no NEW card numbers are stored

   The app fix needs a review and deploy; the collector change takes two minutes. See 'The fix'.
5. 11:10

   HYPOTHESIS

   Where else did these lines go?

   Logs are copied everywhere: the collector's buffer, Loki, exported bug reports, a SIEM, someone's laptop from a docker logs > debug.txt. The incident isn't over when Loki is clean. Inventory every sink.
6. 12:30

   RESOLVED

   App redacts at source; existing lines deleted; breach review opened with security

1. 01Contain: mask card-like numbers in Alloy

   stage.replace replaces each CAPTURE GROUP in the expression with replace. This masks the first 12 digits of any 16-digit number beginning with 4, 5, or 3, keeping the last 4 for support lookups. Add it to the loki.process pipeline from Case 2.1 and restart Alloy.

     stage.replace {
       expression = "\\b([345][0-9]{11})[0-9]{4}\\b"
       replace    = "************"
     }

2. 02Fix at the source: pino redaction

   Even if someone logs req.body again, these paths are replaced. Wildcards cover nested objects. Then switch the scenario off.

   const log = pino({
     level: process.env.LOG_LEVEL ?? "info",
     redact: {
       paths: ["card", "*.card", "req.body.card", "password", "*.password", "req.headers.authorization"],
       censor: "[REDACTED]",
     },
   });

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"pii","enabled":false}'

   ── expected output ──

   {"active":[]}

3. 03Remove what was already stored

   Loki's delete API takes a LogQL selector plus a time range, and requires compactor retention and deletion to be enabled in Loki's config (check the docs for your version). Record the delete request ID in the incident, and verify with the same count query afterwards. For systems without targeted deletion, shorten retention for the affected streams and document the residual window.

   terminalcopy cmd

   $ curl -s -X POST -G 'localhost:3100/loki/api/v1/delete' \

   --data-urlencode 'query={container="shoplite"} |~ "\\b4[0-9]{15}\\b"' \

   --data-urlencode "start=$(date -d '7 days ago' +%s)"

   ── expected output ──

   (204 No Content — deletion scheduled; processed by the compactor)

1. 01Scheduled PII detection

   Run the detection query as a Grafana alert (fire on any match in the last hour) or a nightly job. Add patterns for your own sensitive data: emails, phone numbers, JWTs (eyJ[A-Za-z0-9_-]+\.), cloud keys (AKIA[0-9A-Z]{16}).

   LogQL· Lokicopy

   sum by (container) (count_over_time({container=~".+"} |~ "\\b[345][0-9]{15}\\b|AKIA[0-9A-Z]{16}" [1h])) > 0

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.