“The dashboard says checkout takes 400 ms. Support says customers are waiting 4 seconds.”

1. 14:10

   REPORT

   Three support tickets in an hour: 'payment button spins for ages, then works'

   Intermittent and eventually successful, the hardest kind of report to act on.
2. 14:12

   QUERY

   Check average latency — the number on the existing dashboard

   A histogram's _sum is the total seconds spent; _count is the number of requests. Dividing their rates gives the MEAN latency.

   PromQL· Prometheuscopy

   sum(rate(http_request_duration_seconds_sum{job="shoplite", route="/checkout"}[5m]))
   /
   sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[5m]))

   result

   {}  0.412

3. 14:14

   DEAD END

   '412 ms is a bit high but fine — probably the customers' mobile networks'

   The average blends 90 fast requests (~110 ms) with 10 very slow ones (~3.1 s) into one number that describes NOBODY's actual experience. No customer waited 412 ms.
4. 14:31

   QUERY

   Look at the distribution instead: median and 99th percentile

   histogram_quantile estimates a percentile from bucket counts. The buckets (le = 'less than or equal') must be summed across instances first and KEPT as a label, hence sum by (le).

   PromQL· Prometheuscopy

   histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{job="shoplite", route="/checkout"}[5m])))
   histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="shoplite", route="/checkout"}[5m])))

   result

   p50  {}  0.108
   p99  {}  4.74

5. 14:33

   FINDING

   Median 108 ms, p99 4.7 s: this is a slow TAIL, not a slow service

   Half the requests are fast. A small fraction is dramatically slow. That changes the search completely: look for something that happens occasionally (a specific query, a lock, a cold cache), not for a service that's slow overall.
6. 14:35

   QUERY

   How many checkouts are over 1 second?

   Buckets are cumulative, so the le="1" bucket counts requests that took at most 1s. One minus its share of the total is the fraction slower than 1s. This number is exact, not estimated.

   PromQL· Prometheuscopy

   1 - (
     sum(rate(http_request_duration_seconds_bucket{job="shoplite", route="/checkout", le="1"}[5m]))
     /
     sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[5m]))
   )

   result

   {}  0.101

7. 14:52

   RESOLVED

   A slow report query sharing the table is moved to a read replica

   In our lab the slow query is pg_sleep(3), a stand-in for any occasionally slow statement. Case 0.4 shows how a trace pinpoints exactly which call in the request is slow.

1. 01Turn the slow query off

   In real life this is where you'd find and fix the slow statement. Case 0.4 shows how to find exactly which call is slow using a trace.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"slow-checkout","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Put a bucket boundary at your SLO threshold

   ShopLite's target is 'checkout under 1 s'. 1 is already a boundary, which is why the 'fraction over 1 s' query was exact. Add boundaries where your decisions live, such as 0.3 for the product page, and remove ones you never use. Each bucket is a separate time series per label combination, so more buckets means more storage.

     buckets: [0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 0.75, 1, 2, 3, 5, 10],

2. 02Chart percentiles per route, never the mean alone

   Use this query in the latency panel of the ShopLite dashboard: p99 for every route, estimated from summed buckets.

   PromQL· Prometheuscopy

   histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket{job="shoplite"}[5m])))

   result

   {route="/checkout"}  0.231
   {route="/products"}  0.047

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.