Metrics in Anger

Four incidents where the metrics were there, but reading them wrong, or collecting them wrong, made things worse.

Chapter 0 got metrics flowing. This chapter is about the ways metrics mislead or break: counters that reset on every deploy, one bad label that takes down Prometheus itself, a CPU graph that says 48% while the service is starved, and a memory leak invisible on the graph everyone was watching.

1. CASE 1.1SEV3grafana dashboard · shoplite-api40 min

   “Right after the 15:00 deploy, the 'checkouts today' panel dropped from 41,207 to 312.”

   The deploy that 'lost' 40,000 orders

2. CASE 1.2SEV1prometheus1 h 50 min

   “Prometheus is OOM-killed every 20 minutes and every dashboard is blank. Nothing in ShopLite changed... except one new metric.”

   One label took down Prometheus

3. CASE 1.3SEV2shoplite-api40 min

   “Latency doubled at noon. CPU is only 48%. Scaling-by-CPU didn't kick in.”

   48% CPU and completely saturated

4. CASE 1.4SEV2shoplite-api1 h 5 min

   “ShopLite restarts every six minutes. The heap graph is perfectly flat.”

   The leak the heap graph couldn't see