“Checkout errors are at 20%. The metric can't tell us what the error actually is.”

1. 16:02

   QUERY

   The error ratio for /checkout jumps to 0.2

   The metric knows THAT requests fail and WHERE (route, status). It can't know WHY. Nobody puts error messages in metric labels (see the concepts).

2. 16:04

   QUERY

   Select ShopLite's log stream and keep only error-level lines

   A LogQL query starts with a STREAM SELECTOR in braces, which chooses streams by their labels (here container, set by Alloy). Then comes a PIPELINE: | json parses each line's JSON into fields, and | level = 50 filters on pino's numeric level (50 = error). Nested fields are flattened with underscores, so err.message becomes err_message.

   LogQL· Lokicopy

   {container="shoplite"} | json | level = 50 | line_format "{{.msg}}: {{.err_message}}"

   result

   2026-09-26 16:04:11  checkout failed: could not obtain lock on row in relation "stock"
   2026-09-26 16:04:11  checkout failed: could not obtain lock on row in relation "stock"
   2026-09-26 16:04:10  checkout failed: could not obtain lock on row in relation "stock"
   ...

3. 16:06

   QUERY

   Is it ONE error or several? Count them by message

   count_over_time(...[5m]) turns log lines into a metric: lines per stream per window. sum by (err_message) groups them by the parsed field. One message accounts for all the errors, so it's a single failure mode.

   LogQL· Lokicopy

   sum by (err_message) (
     count_over_time({container="shoplite"} | json | level = 50 [5m])
   )

   result

   {err_message="could not obtain lock on row in relation \"stock\""}  362

4. 16:09

   HYPOTHESIS

   Postgres error 55P03 is lock_not_available: something is holding locks on stock

   The parsed err_code field says 55P03. In a real system the next step is pg_stat_activity / pg_locks to find the blocking session. In Case 0.1 it was an admin script holding a transaction open.

   LogQL· Lokicopy

   {container="shoplite"} | json | level = 50 | line_format "{{.err_code}} {{.err_type}}"

   result

   55P03 Error
   55P03 Error
   ...

5. 16:20

   RESOLVED

   Blocking session terminated; the errors stop

   This time: 18 minutes from first signal to fix, instead of 47 minutes and a customer tweet.

1. 01Resolve and switch the scenario off

   Real fixes for lock contention: keep transactions short, set lock_timeout so waiters fail fast instead of piling up, and use SELECT ... FOR UPDATE SKIP LOCKED for queue-like access patterns.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01The first alert rule: page on user-facing error ratio

   Case 0.1 needed a customer; this case needed someone running a query. An alert rule runs the query every 15 seconds. for: 2m avoids paging on a single bad scrape. Labels route it, and annotations explain it. Chapter 4 replaces this with SLO burn-rate alerts; for now, a sensible threshold is a big step up from nothing.

   groups:
     - name: shoplite
       rules:
         - alert: CheckoutErrorRatioHigh
           expr: |
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[5m]))
             /
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[5m]))
             > 0.05
           for: 2m
           labels:
             severity: critical
             service: shoplite
           annotations:
             summary: "{{ $value | humanizePercentage }} of checkouts are failing"
             runbook: "Check error logs: {container=\"shoplite\"} | json | level = 50"

   terminalcopy cmd

   $ curl -s -X POST localhost:9090/-/reload && curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | .name + " " + .health'

   ── expected output ──

   CheckoutErrorRatioHigh ok

2. 02Test the alert by replaying the incident

   An alert you've never seen fire is an alert you can't trust. Switch errors on, watch the rule go pending (condition true, for not yet satisfied) and then firing at http://localhost:9090/alerts. Chapter 4 routes it to an actual notification channel. Switch it off again afterwards.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors"}'

   sleep 180 && curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | .labels.alertname + " " + .state'

   ── expected output ──

   CheckoutErrorRatioHigh firing

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.