“Checkout p99 is 2.8 seconds. The DBA swears Postgres is idle.”

1. 11:40

   ALERT

   p99 for /checkout crosses 1 s and stays there

   Unlike Case 0.2, this isn't a tail: p50 is slow too. EVERY checkout is slow.

   PromQL· Prometheuscopy

   histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{job="shoplite", route="/checkout"}[5m])))

   result

   {}  2.71

2. 11:44

   DEAD END

   Hypothesis: the database. Postgres CPU 4%, no slow queries, no locks

   Everyone's first suspect, and a classic time sink. Metrics about ShopLite can only say 'ShopLite's requests are slow'. They can't say which PART of each request is slow.
3. 11:51

   QUERY

   Search Tempo for slow checkout traces

   TraceQL selects spans by attributes inside { }. resource.service.name is the service, span.http.route the route (set by the Express instrumentation), and duration the span's own duration.

   TraceQL· Tempocopy

   { resource.service.name = "shoplite" && span.http.route = "/checkout" && duration > 1s }

   result

   Trace ID                          Start     Root service  Root span        Duration
   4bf92f3577b34da6a3ce929d0e0e4736  11:51:07  shoplite      POST /checkout   2.71s
   9a01c3de66f14b1b8c02e5e8f2a7d913  11:51:07  shoplite      POST /checkout   2.69s
   ...

4. 11:52

   FINDING

   The waterfall: 2.68 s of the 2.71 s is inside the payments service

   Both Postgres queries take 2–4 ms. The outgoing POST to payments takes 2,689 ms, and nearly all of it is the POST /charge span INSIDE payments. The trace crossed a service boundary because ShopLite's fetch sent a traceparent header, which the payments instrumentation continued.

   POST /checkoutshoplite

   2712 ms

   middleware - jsonParsershoplite

   1 ms

   pg.query:SELECT shopliteshoplite

   2 ms

   POST (fetch → payments)shoplite

   2689 ms

   POST /chargepayments

   2681 ms

   pg.query:INSERT shopliteshoplite

   4 ms

5. 11:55

   QUERY

   Jump from the trace to its logs

   The pino instrumentation injected trace_id into every log line written during the request, and the Grafana data sources are linked, so from any span you can open exactly that request's logs, in both services.

   LogQL· Lokicopy

   {container=~"shoplite|payments"} | json | trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"

   result

   11:51:07.002 shoplite  checkout started              card_last4=1111 productId=2
   11:51:09.690 payments  request completed             responseTime=2681 statusCode=200
   11:51:09.712 shoplite  request completed             responseTime=2712 statusCode=201

6. 12:15

   RESOLVED

   The payment provider's degraded region is failed over; /charge back to ~70 ms

   The follow-up for ShopLite itself: a 5-second timeout on a call that normally takes 70 ms is far too generous. Users waited the full slowdown instead of getting a fast, retryable error.

1. 01Fail fast: a timeout that matches reality

   /charge normally takes under 150 ms. A 5-second timeout means a degraded dependency makes every user wait 5 seconds. Set it to about 3–5× the normal p99 so slow calls fail quickly and clearly, return an error the client can retry, and stop tying up ShopLite's connections. The error logs then say what happened, and the error-ratio alert from Case 0.3 fires.

         signal: AbortSignal.timeout(1000), // payments p99 is ~120 ms; fail fast instead of making users wait

2. 02Switch the incident off

   terminalcopy cmd

   $ curl -s -X POST localhost:8081/admin/chaos -H 'content-type: application/json' -d '{"scenario":"slow-payments","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Alert on the DEPENDENCY's latency, not only your own

   Payments exposes the same histogram. A rule on its p99 fires on the real culprit, often before ShopLite's own latency alert, and its name tells on-call where to look first.

         - alert: PaymentsLatencyP99High
           expr: |
             histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="payments", route="/charge"}[5m])))
             > 0.5
           for: 5m
           labels:
             severity: warning
             service: payments
           annotations:
             summary: "payments /charge p99 is {{ $value | humanizeDuration }}"

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.