“Product says reliability is fine at 99.4%. Engineering says we're on fire. Both are looking at the same graph.”

1. Mon

   REPORT

   'Checkout success is 99.4% this month. That's basically 100%.'

   At 180,000 checkouts a month, 0.6% is 1,080 failed payments. Whether that's 'fine' isn't a technical question; it's a product decision that nobody had actually made.
2. Tue

   ACTION

   Write the SLO down: SLI, target, window

   SLI: the proportion of checkout requests that succeed (not 5xx). SLO: 99.5% over a rolling 28 days. ERROR BUDGET: the 0.5% allowed to fail, about 900 checkouts at current traffic. Now '99.4%' has a meaning: we've spent 120% of our budget.
3. Tue

   QUERY

   Encode the SLI as a recording rule

   Recording rules compute an expression on every evaluation and store it as a new series. SLO math re-uses the same ratio over several windows, so recording it once at 5m keeps queries fast and consistent.

   PromQL· Prometheuscopy

   slo:checkout_errors:ratio_rate5m

   result

   {service="shoplite", slo="checkout-availability"}  0.199   (errors scenario on)

4. Wed

   QUERY

   Budget remaining over the SLO window

   Error ratio over the window divided by the allowed ratio (0.005) gives the fraction of budget spent. One minus that is what's left. Negative means the SLO is breached.

   PromQL· Prometheuscopy

   1 - (
     sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[28d]))
     /
     sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[28d]))
   ) / 0.005

   result

   {}  -0.21   → budget exhausted: 121% spent

5. Thu

   RESOLVED

   Error budget policy agreed: while budget is negative, reliability work takes priority over features

   The argument ends because the decision was made in advance, by product and engineering together, and now just gets applied.

1. 01SLI recording rules for several windows

   Burn-rate alerts (Case 4.3) need the error ratio over 5m, 30m, 1h, and 6h. Record each once. The slo label names the objective; the service label keeps Alertmanager routing working.

   # add "slo.rules.yml" to rule_files in prometheus.yml
   groups:
     - name: slo-checkout
       rules:
         - record: slo:checkout_errors:ratio_rate5m
           labels: { service: shoplite, slo: checkout-availability }
           expr: |
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[5m]))
             / sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[5m]))
         - record: slo:checkout_errors:ratio_rate30m
           labels: { service: shoplite, slo: checkout-availability }
           expr: |
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[30m]))
             / sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[30m]))
         - record: slo:checkout_errors:ratio_rate1h
           labels: { service: shoplite, slo: checkout-availability }
           expr: |
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[1h]))
             / sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[1h]))
         - record: slo:checkout_errors:ratio_rate6h
           labels: { service: shoplite, slo: checkout-availability }
           expr: |
             sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout", status=~"5.."}[6h]))
             / sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[6h]))

   terminalcopy cmd

   $ curl -s -X POST localhost:9090/-/reload

   curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Write the SLO document, not just the rules

   One page per SLO: the SLI definition (exact query), the target and window, who owns it, the error budget policy (what happens when it's spent), and when it will be reviewed. Rules without an agreed policy are just another graph.

   # SLO: Checkout availability
   
   SLI:     successful /checkout responses (non-5xx) / all /checkout responses
            (recording rule: slo:checkout_errors:ratio_rate5m)
   Target:  99.5% over a rolling 28 days
   Budget:  0.5% of checkouts (~900 at current traffic)
   Owner:   ShopLite team · reviewed quarterly with Product
   
   Policy:
   - Budget > 50%: normal release cadence.
   - Budget < 25%: every release needs a rollback plan and a canary.
   - Budget exhausted: feature releases pause; the top reliability items
     from the last 28 days' incidents are prioritised until budget recovers.

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.