System 12.22 — Metrics / Monitoring
In one line
Prometheus-style: pull model, label-based time series, alerting, and the golden signals.
Think of it like this
The dashboard in a car: speed, fuel, engine temperature, all visible at a glance, with a warning light that flashes before you actually break down.
Key ideas
- 01
Time-series model: metric{labels} → values per interval; the label (service, endpoint, status) IS the query axis.
- 02
Collection: pull (Prometheus scrapes /metrics) vs push (statsd) — pull suits ephemeral K8s with service discovery.
- 03
Cardinality is everything: a label with 10k values (user_id as label) is a dimension explosion that melts storage.
- 04
Golden signals: latency (p50/p95/p99), traffic (rps), errors (rate), saturation (CPU/queue depth) — the SRE four.
- 05
Derived metrics: RED (rate, errors, duration) per service; USE (utilization, saturation, errors) per resource.
- 06
Alerting: rules over query windows, alertmanager dedupe/route to slack/pager; SLO-based alerting (burn rates) over raw thresholds.
- 07
Storage: TSDB with downsampling/retention (Prometheus ops + remote long-term store like Thanos/Cortex for HA).
- 08
Dashboards: Grafana panels from the same queries — correlations, not just pretty graphs.
Code & diagrams
Metrics are PULLED, not pushed — the collector decides the cadence, services just expose a number.
Explain without notes
Why is label cardinality the silent killer of a metrics system — give the math of one bad label at 10 services.
Practice
Define the RED metrics + SLO burn alert thresholds for the payment service.
Trade-offs
- ↔
Precision (per-fields metrics) vs storage cost; high cardinality metrics need aggregation, not indexing.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can design pull-based metrics with golden signals, SLO alerts, and a Thanos-style long-term store.