Seeing Anything at All

Four incidents that teach the three signals: metrics tell you THAT something is wrong, logs tell you WHAT, traces tell you WHERE.

ShopLite is live and customers are finding its problems before the team does. This chapter starts with no monitoring at all, builds a complete observability lab on your laptop, and then works through three incidents, each of which can only be solved with one specific signal: a percentile metric, a structured log query, or a distributed trace.

Every later chapter reuses this lab. Keep obs-lab/ around.

1. CASE 0.1SEV2shoplite-api47 min, most of it spent guessing

   “A customer tweeted that checkout is broken. Is it? For whom? Since when?”

   Flying blind — build the lab

2. CASE 0.2SEV3shoplite-api · /checkout25 min once the right query was run

   “The dashboard says checkout takes 400 ms. Support says customers are waiting 4 seconds.”

   Averages lie — percentiles and histograms

3. CASE 0.3SEV2shoplite-api · /checkout18 min

   “Checkout errors are at 20%. The metric can't tell us what the error actually is.”

   Metrics say 'failing' — logs say why

4. CASE 0.4SEV2shoplite-api → payments35 min

   “Checkout p99 is 2.8 seconds. The DBA swears Postgres is idle.”

   Slow, but not the database — distributed traces