“ShopLite was completely down for 25 minutes. Not one alert fired. Every rule looked correct.”

1. 22:10

   REPORT

   Customers: the site returns 502 from the load balancer

   The error-ratio and burn-rate alerts are perfectly written. They're just not firing.
2. 22:14

   QUERY

   What does the SLI return right now?

   Nothing: an empty result, not 0, not 1. With the process down, Prometheus has no fresh samples for http_request_duration_seconds_count; after a few minutes those series go STALE and disappear from queries. A comparison on an empty vector is empty, and an empty alert expression means 'not firing'.

   PromQL· Prometheuscopy

   slo:checkout_errors:ratio_rate5m

   result

   Empty query result

3. 22:16

   QUERY

   The one metric that DOES know

   Prometheus records every scrape attempt. up is 0 when the scrape fails.

   PromQL· Prometheuscopy

   up{job="shoplite"}

   result

   {instance="shoplite:8080", job="shoplite"}  0

4. 22:35

   RESOLVED

   Process restarted; ShopLiteDown and absence alerts added and unit-tested

1. 01Alert on the scrape and on absence

   ShopLiteDown pages; the Alertmanager inhibit rule from Case 4.1 then suppresses the noise from everything else about shoplite. ShopLiteMetricsAbsent catches the rarer case where the target itself vanished.

         - alert: ShopLiteDown
           expr: up{job="shoplite"} == 0
           for: 1m
           labels: { severity: critical, service: shoplite }
           annotations:
             summary: "shoplite is down (scrape failing)"
         - alert: ShopLiteMetricsAbsent
           expr: absent(up{job="shoplite"})
           for: 5m
           labels: { severity: critical, service: shoplite }
           annotations:
             summary: "No shoplite scrape target exists — removed from discovery or job renamed"

   terminalcopy cmd

   $ curl -s -X POST localhost:9090/-/reload

   sleep 90 && curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | .labels.alertname + " " + .state'

   ── expected output ──

   ShopLiteDown firing

2. 02Bring it back

   terminalcopy cmd

   $ docker compose start shoplite

   ── expected output ──

   ✔ Container shoplite Started

1. 01Unit-test the rule with promtool

   The test feeds up values minute by minute (1, 1, then 0 from minute 2) and asserts that at minute 5 the alert is firing with exactly these labels and annotations. Add a test like this for every paging alert and run promtool test rules in CI.

   rule_files: [alerts.yml]
   evaluation_interval: 1m
   
   tests:
     - interval: 1m
       input_series:
         - series: 'up{job="shoplite", instance="shoplite:8080"}'
           values: '1 1 0 0 0 0 0'
       alert_rule_test:
         - eval_time: 1m
           alertname: ShopLiteDown
           exp_alerts: []
         - eval_time: 5m
           alertname: ShopLiteDown
           exp_alerts:
             - exp_labels:
                 severity: critical
                 service: shoplite
                 job: shoplite
                 instance: "shoplite:8080"
               exp_annotations:
                 summary: "shoplite is down (scrape failing)"

   terminalcopy cmd

   $ docker compose exec prometheus promtool test rules /etc/prometheus/alerts.test.yml

   ── expected output ──

   Unit Testing: /etc/prometheus/alerts.test.yml

   SUCCESS

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.