“On-call got paged 64 times last night. None of the pages needed a human. At 06:10 a real one was acknowledged and ignored.”

1. 08:30

   QUERY

   Which alerts fired, and how often?

   Prometheus exposes its own alert state as the ALERTS series. Counting firing samples per alert over the night ranks the noisiest ones.

   PromQL· Prometheuscopy

   sort_desc(count by (alertname) (count_over_time(ALERTS{alertstate="firing"}[12h])))

   result

   {alertname="HighCPU"}                       412
   {alertname="ShopLiteRestartingFrequently"}  96
   {alertname="PodMemoryAbove80Percent"}       88
   {alertname="CheckoutErrorRatioHigh"}         7
   {alertname="DiskWillFillIn4Hours"}           3

2. 08:40

   FINDING

   The noise was about CAUSES, not SYMPTOMS

   HighCPU fired during nightly batch jobs, when high CPU is the whole point. The memory alert fired because the service runs close to its limit by design. Restart alerts fired during a planned node rotation. None of them meant a user was affected. CheckoutErrorRatioHigh, the one that did, looked exactly like the others and got the same tired acknowledgment.
3. 08:55

   FINDING

   Every alert went to the same pager with the same urgency

   No severity routing, no grouping, no inhibition. When ShopLite went down, eleven different alerts fired about it separately, one per metric.
4. Next week

   RESOLVED

   Alerts triaged: 4 page, 9 ticket, 23 deleted; Alertmanager routes by severity

1. 01Add Alertmanager and a webhook logger to the lab

   The logger is an echo server that prints every notification it receives to stdout. Alloy ships that to Loki, so you can see exactly what 'got paged' with {container=~".*alert-logger.*"}. Tell Prometheus where Alertmanager is.

   # docker-compose.yml
     alertmanager:
       image: prom/alertmanager:v0.28.1
       command: --config.file=/etc/alertmanager/alertmanager.yml
       volumes: ["./alertmanager:/etc/alertmanager:ro"]
       ports: ["9093:9093"]
     alert-logger:
       image: mendhak/http-https-echo:37
   
   # prometheus/prometheus.yml
   alerting:
     alertmanagers:
       - static_configs: [{ targets: ["alertmanager:9093"] }]

2. 02Route by severity, group by service, inhibit when the service is down

   Default route: tickets. Critical alerts go to the pager. Alerts for the same service are grouped into one notification. When ShopLiteDown fires, every other shoplite alert is inhibited, since one clear page beats eleven vague ones.

   route:
     receiver: ticket
     group_by: [alertname, service]
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 4h
     routes:
       - matchers: [severity="critical"]
         receiver: pager
         group_by: [service]
   
   inhibit_rules:
     - source_matchers: [alertname="ShopLiteDown"]
       target_matchers: [service="shoplite", alertname!="ShopLiteDown"]
       equal: [service]
   
   receivers:
     - name: pager
       webhook_configs: [{ url: "http://alert-logger:8080/pager" }]
     - name: ticket
       webhook_configs: [{ url: "http://alert-logger:8080/ticket" }]

   terminalcopy cmd

   $ docker compose exec alertmanager amtool config routes test \

   --config.file=/etc/alertmanager/alertmanager.yml severity=critical service=shoplite

   ── expected output ──

   pager

3. 03Re-label the rules you keep; delete the rest

   severity: critical only for user-facing symptoms: CheckoutErrorRatioHigh (burn-rate in Case 4.3) and ShopLiteDown (Case 4.4). Warnings like memory prediction, event-loop lag, and retry amplification become severity: warning and go to tickets. Delete HighCPU.

4. 04Silence planned work

   Before a maintenance window, silence matching alerts for a fixed duration with a comment. Everyone can see who silenced what and why in the Alertmanager UI.

   terminalcopy cmd

   $ docker compose exec alertmanager amtool silence add service=shoplite \

   --alertmanager.url=http://localhost:9093 --duration=1h --comment="node rotation, ticket OPS-412"

   ── expected output ──

   b7e3d1a2-4c5f-4e8a-9d0b-1f2e3a4b5c6d

1. 01Measure your paging load

   Track notifications per receiver per week, and review them in the on-call handover. A common target is fewer than two pages per on-call shift. If it's higher, delete or demote alerts before adding new ones.

   PromQL· Prometheuscopy

   sum by (receiver) (increase(alertmanager_notifications_total{integration="webhook"}[7d]))

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.