Guide G7 · DevOps path

Incident Management and Reliability Metrics

The incident lifecycle from detection to postmortem: severity levels, on-call and escalation, incident command roles, runbooks vs playbooks, root cause analysis, corrective actions, and the metrics that measure it all (MTTD, MTTA, MTTR, MTBF, change failure rate).

Intermediate 45 min

Start here

The mental model

An incident is a fire; incident management is the fire brigade's procedure. Nobody improvises who drives the truck or who talks to the press: roles, radio channels, and checklists are decided in advance, so under stress people follow the procedure instead of arguing. Afterwards, the brigade studies how the fire started and what would stop the next one, without blaming the person who left the stove on.

The goal during an incident is to RESTORE SERVICE (mitigate), not to find the root cause. Rollback first, understand later. The goal after the incident is to LEARN, so the same class of failure doesn't happen again.

Go deeper

How it works inside

01The lifecycle

DETECT (an alert, ideally SLO-based, or a customer report), TRIAGE (how bad? assign a severity), RESPOND (open an incident channel, assign an incident commander, mitigate: roll back, fail over, scale, disable a feature flag), COMMUNICATE (status page, stakeholders, at a regular cadence), RESOLVE (service healthy and stable), then LEARN (blameless postmortem, action items tracked to completion).

The lifecyclediagram

Rendering diagram…

02Severity and escalation

A SEVERITY scale makes response proportional and consistent. A typical one: SEV1 (critical: major customer-facing outage or data loss; all hands, exec updates, 24/7), SEV2 (major: significant degradation or a key feature down; on-call plus owners, business hours escalation), SEV3 (minor: limited impact or a workaround exists; handled in normal working hours). Define them by CUSTOMER IMPACT, not by which component broke. When unsure, declare higher: downgrading is cheap, a slow response isn't.

ON-CALL rotations give each service a primary and secondary responder. An ESCALATION POLICY defines what happens if a page isn't acknowledged (e.g. after 5 minutes page the secondary, after 15 the engineering manager). Tools: PagerDuty, Opsgenie, incident.io, Grafana OnCall; Alertmanager routes alerts to them by team and severity.

03Roles: incident command

For anything beyond a small incident, separate the roles (SRE course, incident command): the INCIDENT COMMANDER (IC) coordinates, makes decisions, and keeps the big picture, and deliberately does NOT debug. OPERATIONS/subject-matter experts investigate and apply fixes. The COMMUNICATIONS lead handles status pages and stakeholder updates. A SCRIBE keeps the timeline. In a small team one person may hold two roles, but the IC must still own coordination.

04Runbooks vs playbooks

A RUNBOOK is a specific, step-by-step procedure for one alert or task: 'CheckoutLatencyBurnFast: 1. open this dashboard, 2. check payments p99, 3. if > 1s, enable the fallback flag...'. Link it from the alert annotation so on-call gets it in the page. A PLAYBOOK is broader guidance for a type of situation: 'how we handle a security incident', 'how we run a SEV1', 'region failover'. Good runbooks are tested, dated, and owned, and the best ones are gradually automated (auto-remediation).

05Root cause analysis and corrective actions

Complex systems rarely have a single root cause; there are CONTRIBUTING FACTORS (a risky change, a missing alert, a timeout set too high, an unclear runbook). Techniques: the FIVE WHYS (keep asking why until you reach something you can change in the system, not a person), a TIMELINE reconstruction, and fishbone (Ishikawa) diagrams for complex cases.

CORRECTIVE ACTIONS should be specific, owned, and dated, and spread across prevention (fix the bug, add a test or policy), detection (a better alert, faster detection), and mitigation (a feature flag, a runbook, automated rollback). 'Be more careful' is not an action. Track completion; a postmortem whose actions never land is theatre. BLAMELESSNESS: focus on how the system allowed the mistake, because people who fear blame hide information (SRE course, blameless postmortem).

06The metrics

MTTD (mean time to detect): failure start → detection. Improved by SLO/burn-rate alerts and synthetic checks (Observability course). MTTA (mean time to acknowledge): page → a human responds; shows on-call health. MTTR (mean time to restore/recover): failure start (or detection) → service restored; improved by runbooks, fast rollback, feature flags, and canaries. MTBF (mean time between failures): average uptime between incidents; improved by prevention. CHANGE FAILURE RATE: share of deployments causing an incident or rollback (a DORA metric). Always state your definitions (e.g. does MTTR start at failure or detection?).

Treat them as TRENDS for learning, not targets for individuals: averages hide the long tail, so look at the distribution and the worst incidents too. Availability relates to them roughly as MTBF / (MTBF + MTTR): you get more reliable either by failing less often or by recovering faster, and recovering faster is usually cheaper.

The metricsdiagram

Rendering diagram…

Do it

Hands-on lab

Define your severity matrix

Write it before you need it, agree it with product and support, and publish it where on-call can find it.

docs/incident/severity.mdwhole filemarkdown

| Sev  | Customer impact                                      | Response                        | Updates        |
|------|------------------------------------------------------|---------------------------------|----------------|
| SEV1 | Checkout/login down, data loss, security breach      | Page now, IC + comms, 24/7      | Every 30 min   |
| SEV2 | Major feature degraded, > 5% errors, one region down | Page on-call, owners join       | Every 60 min   |
| SEV3 | Minor feature broken, workaround exists              | Ticket, next business day       | On resolution  |

Route alerts by team and severity

Alertmanager sends pages to the owning team's on-call tool and lower severities to chat, with the runbook link from the alert annotation (Observability course).

alertmanager.yamlwhole fileyaml

route:
  receiver: chat-default
  group_by: [alertname, service]
  routes:
    - matchers: [severity="page", team="checkout"]
      receiver: checkout-pager
      repeat_interval: 30m
    - matchers: [severity="ticket"]
      receiver: chat-default
receivers:
  - name: checkout-pager
    pagerduty_configs:
      - routing_key: <from-secret>
        description: '{{ .CommonAnnotations.summary }}'
        links: [{ href: '{{ .CommonAnnotations.runbook_url }}', text: Runbook }]
  - name: chat-default
    slack_configs: [{ channel: '#alerts', api_url: <from-secret> }]

A runbook and a postmortem template

Keep them in Git next to the service (or in TechDocs, Platform course Mission 1.3) so they're reviewed and versioned.

runbooks/checkout-latency.mdwhole filemarkdown

# Runbook: CheckoutLatencyBurnFast
Owner: team-checkout · Last tested: 2026-09-12
1. Open the Checkout SLO dashboard; confirm p95 > 400 ms and error budget burning.
2. Check the latest deploy (Argo CD history). Deployed < 30 min ago? → roll back (git revert) and re-check.
3. Check payments p99 on the dependency panel. > 1 s? → enable flag `payments.fallback-queue` (orders queued, charged later).
4. Check DB: active connections and top queries (Stateful course Unit 1.2). Pool exhausted? → scale PgBouncer pool per runbook DB-3.
5. Still burning after 15 min → declare SEV2, page IC, post status update.

# Postmortem template
- Summary · Impact (users, duration, SLO budget used) · Severity
- Timeline (UTC) · Detection (how, MTTD) · Response (MTTA, MTTR)
- Contributing factors (5 whys) · What went well / what didn't
- Action items: prevention · detection · mitigation (owner, due date, ticket)

Compute the metrics from your incident log

A small script over an exported incident list gives the trend each quarter. Report medians alongside means because one long incident skews the average.

incident_metrics.pywhole filepython

import csv, statistics as st
from datetime import datetime as dt

rows = list(csv.DictReader(open("incidents.csv")))   # started, detected, acked, restored (ISO times)
mins = lambda a, b: (dt.fromisoformat(b) - dt.fromisoformat(a)).total_seconds() / 60
ttd = [mins(r["started"], r["detected"]) for r in rows]
tta = [mins(r["detected"], r["acked"]) for r in rows]
ttr = [mins(r["started"], r["restored"]) for r in rows]
starts = sorted(dt.fromisoformat(r["started"]) for r in rows)
gaps_h = [(b - a).total_seconds() / 3600 for a, b in zip(starts, starts[1:])]

for name, v in [("MTTD", ttd), ("MTTA", tta), ("MTTR", ttr)]:
    print(f"{name}: mean {st.mean(v):.0f} min, median {st.median(v):.0f} min")
print(f"MTBF: {st.mean(gaps_h):.0f} h between incidents ({len(rows)} incidents)")

terminal

$ python incident_metrics.py

── expected output ──

MTTD: mean 9 min, median 4 min

MTTA: mean 3 min, median 2 min

MTTR: mean 52 min, median 31 min

MTBF: 214 h between incidents (11 incidents)

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
Escalation timeout	tool-specific	How long before an unacknowledged page escalates.	5 min to secondary, 15 min to manager for SEV1/2 pages.
Alert severity labels	—	Which alerts page vs ticket.	Page only on symptoms needing action now (SLO burn); everything else tickets or chat.
Status update cadence	—	How often stakeholders hear from you.	Fixed interval by severity, even if the update is 'no change'.

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Everyone debugging, nobody leading

A SEV1 checkout outage: eight engineers join the call, all investigating different theories. Support asks for an update three times; nobody answers. It takes 95 minutes to roll back a deploy that went out just before the outage.

terminal

$ grep -c 'anyone looking at' incident-channel-export.txt; grep -m1 'rolled back' incident-channel-export.txt

── what you'll see ──

14:41 ravi: rolled back checkout to previous version, errors dropping

Drill #2

Customers found it first

Search has returned empty results for 40 minutes. Support tickets reveal it; no alert fired.

terminal

$ promtool query instant http://prometheus:9090 'sum(rate(http_server_requests_seconds_count{service="search",status=~"5.."}[5m]))'

── what you'll see ──

0 # no errors: the service returned 200 with empty results

Decide

Incident metrics at a glance

Metric	Measures	From → to	Improve with
MTTD	Detection speed	Failure start → alert/detection	SLO burn-rate alerts, synthetic checks, correctness SLIs
MTTA	Responder speed	Page → acknowledgement	Healthy on-call rota, sane alert volume, escalation policy
MTTR	Recovery speed	Failure start (or detection) → restored	Runbooks, fast rollback, feature flags, canaries, IC practice
MTBF	How often things fail	End of one incident → start of the next	Testing, safer changes, redundancy, chaos engineering
Change failure rate	Delivery stability	Deploys causing incidents ÷ all deploys	Tests, canaries, smaller changes (DORA)

The bigger picture

Connects to

System Design · Reliability

Failover, redundancy, replication and health checks — the mechanisms that make downtime an exception, not a feature.

SRE · Your first page

Practise triage and mitigation in the on-call simulator.

SRE · Incident command

Running a SEV1 with roles.

SRE · Blameless postmortem

Writing the learning review.

Observability · Burn-rate alerts

The alerts that cut MTTD.

Platform · Measuring the platform

Change failure rate and recovery time as DORA metrics.

Prove it

Interview questions

Walk me through how you handle a production incident.

What are MTTD, MTTR, and MTBF, and which would you improve first?

Runbook vs playbook?

Command Palette

The mental model

How it works inside

01The lifecycle

02Severity and escalation

03Roles: incident command

04Runbooks vs playbooks

05Root cause analysis and corrective actions

06The metrics

Hands-on lab

Define your severity matrix

Route alerts by team and severity

A runbook and a postmortem template

Compute the metrics from your incident log

Knobs that matter

Failure drills

Everyone debugging, nobody leading

Customers found it first

Incident metrics at a glance

Connects to

Interview questions