“The 5% threshold paged us for a 90-second blip — and stayed silent while a 1% error rate ate the whole month's budget in three days.”

1. Day 1

   FINDING

   A deploy causes 1% errors — below the 5% threshold

   The static alert never fires. 1% is twice the SLO's allowed 0.5%, which quietly burns budget at 2× speed.

2. Day 3

   REPORT

   The SLO dashboard shows the budget gone
3. Day 5

   ALERT

   Meanwhile: a 90-second spike to 30% errors during a DB failover pages the on-call at 03:12

   It resolved itself before anyone opened a laptop, having used about 0.1% of the monthly budget. The static threshold treats a huge but tiny event as urgent and a small but relentless one as fine. Both are backwards.
4. Day 6

   RESOLVED

   Replaced with multi-window burn-rate alerts

1. 01Burn-rate alert rules on top of the recorded SLIs

   0.005 is the budget (1 − 0.995). The fast and medium burns page; the slow burn creates a ticket. The 3-day window uses increase-based math for brevity; in production, record a 3d ratio too.

     - name: slo-checkout-alerts
       rules:
         - alert: CheckoutErrorBudgetFastBurn
           expr: |
             slo:checkout_errors:ratio_rate1h  > (14.4 * 0.005)
             and
             slo:checkout_errors:ratio_rate5m  > (14.4 * 0.005)
           labels: { severity: critical, service: shoplite, slo: checkout-availability }
           annotations:
             summary: "Checkout burning error budget at {{ $value | humanizePercentage }} errors (>14.4× budget) — ~2% of monthly budget per hour"
         - alert: CheckoutErrorBudgetMediumBurn
           expr: |
             slo:checkout_errors:ratio_rate6h  > (6 * 0.005)
             and
             slo:checkout_errors:ratio_rate30m > (6 * 0.005)
           labels: { severity: critical, service: shoplite, slo: checkout-availability }
           annotations:
             summary: "Checkout burning error budget at >6× — budget exhausted in under 5 days at this rate"
         - alert: CheckoutErrorBudgetSlowBurn
           expr: |
             (
               sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[3d]))
               / sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[3d]))
             ) > 0.005
             and
             slo:checkout_errors:ratio_rate6h > 0.005
           labels: { severity: warning, service: shoplite, slo: checkout-availability }
           annotations:
             summary: "Checkout error rate above SLO for days — on track to breach"

   terminalcopy cmd

   $ curl -s -X POST localhost:9090/-/reload

   sleep 150 && curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | .labels.alertname + " " + .state'

   ── expected output ──

   CheckoutErrorBudgetFastBurn firing

   CheckoutErrorRatioHigh firing

2. 02Retire the static threshold

   Delete CheckoutErrorRatioHigh from alerts.yml. The burn-rate rules cover it with better precision and recall. Then switch the scenario off and watch the fast-burn alert RESOLVE within ~5 minutes, thanks to the short window.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Apply the same pattern to latency

   A latency SLO ('99% of checkouts under 1 s') gets the same treatment, with the SLI from the le="1" bucket. Once you have the pattern, every SLO gets alerts for free. Tools like Sloth or Pyrra generate these rules from a short SLO spec.

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.