Casebook · 6 chapters · 24 incidents
Every incident, on one board
Each card is a real class of production incident. Work them in order: every case assumes the tools and habits from the ones before it.
Seeing Anything at All
Four incidents that teach the three signals: metrics tell you THAT something is wrong, logs tell you WHAT, traces tell you WHERE.
“A customer tweeted that checkout is broken. Is it? For whom? Since when?”
Flying blind — build the lab
“The dashboard says checkout takes 400 ms. Support says customers are waiting 4 seconds.”
Averages lie — percentiles and histograms
“Checkout errors are at 20%. The metric can't tell us what the error actually is.”
Metrics say 'failing' — logs say why
“Checkout p99 is 2.8 seconds. The DBA swears Postgres is idle.”
Slow, but not the database — distributed traces
Metrics in Anger
Four incidents where the metrics were there, but reading them wrong, or collecting them wrong, made things worse.
“Right after the 15:00 deploy, the 'checkouts today' panel dropped from 41,207 to 312.”
The deploy that 'lost' 40,000 orders
“Prometheus is OOM-killed every 20 minutes and every dashboard is blank. Nothing in ShopLite changed... except one new metric.”
One label took down Prometheus
“Latency doubled at noon. CPU is only 48%. Scaling-by-CPU didn't kick in.”
48% CPU and completely saturated
“ShopLite restarts every six minutes. The heap graph is perfectly flat.”
The leak the heap graph couldn't see
Logs at Scale
Four incidents about logs as a system in their own right: their volume, what they must never contain, how to line them up with change, and how to parse the ones you didn't write.
“Log storage grew 40 GB overnight and every Loki query now times out. Nobody changed the logging config.”
One log line, 40 GB a day
“Security found full card numbers in our logs. Twelve teams have read access. How many, since when, and where else?”
Card numbers in the log store
“Checkout errors started 'sometime this afternoon'. There were three deploys. Which one?”
Was it the deploy? Diffing before and after
“Some checkouts are slow and nobody can say which query. The answer was already in Postgres's own logs.”
The database was telling us all along
Tracing Deep Dive
Four incidents that only make sense as a trace: death by a thousand queries, retries that amplify an outage, a trace that silently breaks at a queue, and the trace you needed but didn't keep.
“The product page gets slower every time the catalogue team adds products. No single query is slow.”
Death by a thousand queries
“Payments got slow for two minutes. Then it stayed down for forty — and got four times our normal traffic.”
The retries that kept payments down
“Some receipt emails arrive 20 minutes late. Every checkout trace ends at 'insert into jobs' — and then nothing.”
The trace that stops at the queue
“Support has an order ID and a timestamp for a failed checkout. There's no trace for it — we only keep 10%.”
The trace we needed was sampled away
Alerts & SLOs
Four incidents about the alerting itself: pages nobody should get, reliability nobody could agree on, thresholds that are both too noisy and too slow, and the alert that couldn't fire.
“On-call got paged 64 times last night. None of the pages needed a human. At 06:10 a real one was acknowledged and ignored.”
64 pages, zero actions
“Product says reliability is fine at 99.4%. Engineering says we're on fire. Both are looking at the same graph.”
'99.4% is fine' — or is it?
“The 5% threshold paged us for a 90-second blip — and stayed silent while a 1% error rate ate the whole month's budget in three days.”
Too noisy and too slow at the same time
“ShopLite was completely down for 25 minutes. Not one alert fired. Every rule looked correct.”
The alert that couldn't fire
Production Observability
Four incidents about the observability platform itself: the monitor that silently stopped monitoring, the data you needed but no longer had, the outage only an outsider could see, and the dashboard nobody could find.
“Last Tuesday's new alert never fired. It was never loaded — the config reload had been failing silently for six days.”
Who monitors the monitor?
“Finance wants Q3's checkout latency for the quarterly review. Prometheus keeps 15 days. And last week's restart left a 20-minute hole.”
The data we no longer had
“Every dashboard was green for 50 minutes while every customer saw a certificate error.”
Green inside, broken outside
“During the outage, three engineers opened three different 'ShopLite' dashboards. Two were broken, and the third used the old metric names.”
214 dashboards and none of them right