“Last Tuesday's new alert never fired. It was never loaded — the config reload had been failing silently for six days.”

1. Tue

   ACTION

   A teammate adds a new alert and pushes the config; the deploy script runs curl -X POST /-/reload and ignores the response
2. Mon

   REPORT

   Review: the alert added last Tuesday should have fired twice. It didn't exist in Prometheus at all.
3. Mon

   QUERY

   Prometheus reports its own config health

   Prometheus is itself instrumented. prometheus_config_last_reload_successful has been 0 since Tuesday; ..._success_timestamp_seconds shows exactly when the last good reload happened.

   PromQL· Prometheuscopy

   prometheus_config_last_reload_successful
   (time() - prometheus_config_last_reload_success_timestamp_seconds) / 86400

   result

   {instance="localhost:9090", job="prometheus"}  0
   {instance="localhost:9090", job="prometheus"}  6.2   (days since the last successful reload)

4. Mon

   QUERY

   …and whether rule evaluation or notification is failing

   PromQL· Prometheuscopy

   sum by (rule_group) (rate(prometheus_rule_evaluation_failures_total[5m])) > 0
   or
   rate(prometheus_notifications_errors_total[5m]) > 0

   result

   Empty query result   (healthy — only the reload was broken)

5. Mon

   RESOLVED

   Config fixed; meta-alerts, a Watchdog heartbeat, and CI validation added

1. 01Meta-alerts on Prometheus and Alertmanager

     - name: meta
       rules:
         - alert: PrometheusConfigReloadFailed
           expr: prometheus_config_last_reload_successful == 0
           for: 10m
           labels: { severity: critical, service: observability }
           annotations:
             summary: "Prometheus config reload failing — running on stale rules"
         - alert: PrometheusRuleEvaluationFailures
           expr: sum by (rule_group) (rate(prometheus_rule_evaluation_failures_total[5m])) > 0
           for: 10m
           labels: { severity: warning, service: observability }
         - alert: AlertmanagerNotificationsFailing
           expr: sum by (integration) (rate(alertmanager_notifications_failed_total[5m])) > 0
           for: 10m
           labels: { severity: critical, service: observability }
         - alert: Watchdog
           expr: vector(1)
           labels: { severity: none, service: observability }
           annotations:
             summary: "Always firing. If this stops arriving at the heartbeat service, alerting is broken."

2. 02Route the Watchdog to an external heartbeat

   Add a scrape job for alertmanager:9093 too, so its metrics exist. The Watchdog route repeats every minute; the heartbeat service is configured to alert if it misses, say, 5 minutes of check-ins. In the lab, the alert-logger stands in for it.

     routes:
       - matchers: [alertname="Watchdog"]
         receiver: heartbeat
         group_wait: 0s
         group_interval: 1m
         repeat_interval: 1m
       # ...existing critical → pager route
   
   receivers:
     - name: heartbeat
       webhook_configs: [{ url: "http://alert-logger:8080/heartbeat" }]  # real life: https://hc-ping.com/<uuid>

1. 01Validate before you reload — in CI

   promtool check config validates prometheus.yml and every referenced rules file; amtool check-config does the same for Alertmanager. Run both, plus the promtool test rules from Case 4.4, on every pull request, and make the deploy script fail if /-/reload doesn't return 200.

   terminalcopy cmd

   $ docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml

   ── expected output ──

   Checking /etc/prometheus/prometheus.yml

   SUCCESS: 2 rule files found

   SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

    

   Checking /etc/prometheus/alerts.yml

   SUCCESS: 21 rules found

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.