“Checkout errors started 'sometime this afternoon'. There were three deploys. Which one?”

1. 15:48

   ALERT

   CheckoutErrorRatioHigh firing

   Three deploys went out today: 11:20, 14:02, 15:31. Rolling back all three is slow and risky; rolling back the wrong one wastes the whole incident.
2. 15:50

   QUERY

   When exactly did errors start — and when did processes restart?

   Plot the error ratio and the process start times together. A restart that lines up with the step change in errors is the prime suspect. In Grafana, show restarts as annotations (Case 1.1).

   PromQL· Prometheuscopy

   timestamp(changes(process_start_time_seconds{job="shoplite"}[1m]) > 0)

   result

   {instance="shoplite:8080", job="shoplite"}  1758899460   → 15:31:00

3. 15:53

   QUERY

   Did the NEW version start logging errors the old one never logged?

   offset shifts a query back in time. unless keeps only series on the left that have NO match on the right, so this returns error messages that exist now but didn't exist an hour ago: errors introduced since then.

   LogQL· Lokicopy

   sum by (err_message) (count_over_time({container="shoplite"} | json | level = 50 [15m]))
   unless
   sum by (err_message) (count_over_time({container="shoplite"} | json | level = 50 [15m] offset 1h))

   result

   {err_message="could not obtain lock on row in relation \"stock\""}  274

4. 15:56

   QUERY

   Compare the same metric now vs one hour ago

   offset in PromQL gives the same shape of answer: a before/after comparison without eyeballing graphs.

   PromQL· Prometheuscopy

   sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[10m]))
   -
   sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[10m] offset 1h))

   result

   {}  1.19   (errors/s higher than an hour ago)

5. 16:10

   RESOLVED

   The 15:31 release is rolled back; new error message disappears

1. 01Roll back — here, switch the failure off

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01A build_info metric

   Gauge value is always 1; the information is in the labels. Join it onto other metrics with * on (instance) group_left(version) shoplite_build_info to split anything by version.

   const buildInfo = new client.Gauge({
     name: "shoplite_build_info",
     help: "Build information",
     labelNames: ["version", "commit"],
   });
   buildInfo.set({ version: process.env.APP_VERSION ?? "dev", commit: process.env.GIT_SHA ?? "unknown" }, 1);
   log.info({ version: process.env.APP_VERSION, commit: process.env.GIT_SHA }, "starting");

2. 02Error ratio split by version

   During a canary or rolling deploy, this shows immediately whether the new version is worse than the old one.

   PromQL· Prometheuscopy

   sum by (version) (
     rate(http_request_duration_seconds_count{job="shoplite", status=~"5.."}[5m])
     * on (instance) group_left(version) shoplite_build_info
   )
   /
   sum by (version) (
     rate(http_request_duration_seconds_count{job="shoplite"}[5m])
     * on (instance) group_left(version) shoplite_build_info
   )

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.