Logs at Scale

Four incidents about logs as a system in their own right: their volume, what they must never contain, how to line them up with change, and how to parse the ones you didn't write.

Chapter 0 made logs structured and searchable. Now they grow up: a single noisy line multiplies the logging bill, a debug statement puts card numbers in storage that twelve teams can read, 'did the deploy do it?' needs an answer in minutes, and the most useful log in the building turns out to be Postgres's own.

1. CASE 2.1SEV3loki · shoplite-api3 h

   “Log storage grew 40 GB overnight and every Loki query now times out. Nobody changed the logging config.”

   One log line, 40 GB a day

2. CASE 2.2SEV1shoplite-api · loki2 h to contain; days of follow-up

   “Security found full card numbers in our logs. Twelve teams have read access. How many, since when, and where else?”

   Card numbers in the log store

3. CASE 2.3SEV2shoplite-api22 min

   “Checkout errors started 'sometime this afternoon'. There were three deploys. Which one?”

   Was it the deploy? Diffing before and after

4. CASE 2.4SEV3postgres30 min

   “Some checkouts are slow and nobody can say which query. The answer was already in Postgres's own logs.”

   The database was telling us all along