“Right after the 15:00 deploy, the 'checkouts today' panel dropped from 41,207 to 312.”

1. 15:03

   REPORT

   'Did we just lose forty thousand orders??'

   The panel's query is the raw counter value, the same thing Case 0.1 warned about.

   PromQL· Prometheuscopy

   sum(http_request_duration_seconds_count{job="shoplite", route="/checkout", status="201"})

   result

   {}  312

2. 15:06

   HYPOTHESIS

   Orders table in Postgres: 41,519 rows, still growing

   The business is fine. The NUMBER on the dashboard is wrong, so the question becomes: what does this metric actually measure?
3. 15:14

   FINDING

   The counter lives in the process — and the process was replaced

   Prometheus client libraries keep counters in memory. A new process starts at 0. process_start_time_seconds (a default metric) changed at 15:00, and so did the counter.

   PromQL· Prometheuscopy

   changes(process_start_time_seconds{job="shoplite"}[1h])

   result

   {instance="shoplite:8080", job="shoplite"}  1

4. 15:20

   QUERY

   increase() over the day — resets handled

   increase(counter[24h]) computes how much the counter went up over the window. When it sees a value drop, it assumes a reset and adds the pre-reset value. The dashboard shows the right number again.

   PromQL· Prometheuscopy

   sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout", status="201"}[24h]))

   result

   {}  41519.6

5. 15:43

   RESOLVED

   Panel fixed; escalation stood down

   Note the .6: increase extrapolates to the window edges, so it's an estimate, not an exact count. The concepts section explains why that's fine for dashboards but not for billing.

1. 01Rewrite the panel with increase()

   For a 'today' panel, set the range to the dashboard's time range ($__range in Grafana) so the number always matches the selected period.

   PromQL· Prometheuscopy

   sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout", status="201"}[$__range]))

1. 01Make restarts visible on every dashboard

   Add this as a Grafana annotation query. A vertical line at every process start makes 'that change happened at the deploy' obvious, and explains any step change in a graph at a glance.

   PromQL· Prometheuscopy

   changes(process_start_time_seconds{job="shoplite"}[1m]) > 0

2. 02Alert on crash-looping, which is resets you didn't plan

   A deploy restarts once. A crash loop restarts constantly. restart: unless-stopped in the lab (and ECS or Kubernetes in production) hides crashes by restarting, so alert on the restart RATE.

         - alert: ShopLiteRestartingFrequently
           expr: changes(process_start_time_seconds{job="shoplite"}[15m]) > 3
           labels: { severity: warning, service: shoplite }
           annotations:
             summary: "shoplite restarted {{ $value }} times in 15 minutes"

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.