Tracing Deep Dive

Four incidents that only make sense as a trace: death by a thousand queries, retries that amplify an outage, a trace that silently breaks at a queue, and the trace you needed but didn't keep.

Case 0.4 introduced traces as 'where did the time go?'. This chapter goes further: using trace SHAPE to spot patterns metrics can't see, generating metrics from spans, carrying context through asynchronous work, and deciding which traces are worth keeping.

1. CASE 3.1SEV3shoplite-api · GET /products1 h (slow-burn)

   “The product page gets slower every time the catalogue team adds products. No single query is slow.”

   Death by a thousand queries

2. CASE 3.2SEV1shoplite-api → payments40 min

   “Payments got slow for two minutes. Then it stayed down for forty — and got four times our normal traffic.”

   The retries that kept payments down

3. CASE 3.3SEV3shoplite-api · receipt worker2 h

   “Some receipt emails arrive 20 minutes late. Every checkout trace ends at 'insert into jobs' — and then nothing.”

   The trace that stops at the queue

4. CASE 3.4SEV3tracing pipelineHalf a day to redesign sampling

   “Support has an order ID and a timestamp for a failed checkout. There's no trace for it — we only keep 10%.”

   The trace we needed was sampled away