Case 1.3·Metrics in AngerSEV2

“Latency doubled at noon. CPU is only 48%. Scaling-by-CPU didn't kick in.”

case name: 48% CPU and completely saturated

Service: shoplite-api
Impact: p99 of /products went from 50 ms to 900 ms for 40 minutes
Detected by: p99 latency panel
Time to resolve: 40 min

Skills you'll use on this case

USE methodCPU usage vs limitprocess_cpu_seconds_totalevent-loop lagthrottling

00 It starts

Reproduce this incident in your lab, then work the case alongside the timeline below. Try each query yourself before reading its result.

terminal

$ # 1. give shoplite a CPU limit (add to its service in docker-compose.yml): cpus: "0.5"

docker compose up -d shoplite

# 2. make each /products request burn 50 ms of CPU

curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"cpu-burn"}'

── lab output ──

{"active":["cpu-burn"]}

Real systems always have limits: ECS task CPU, Kubernetes resources.limits.cpu, a VM's vCPUs.

01 The investigation

12:04
ALERT
p99 latency on /products jumps from 50 ms to ~900 ms
/products p99 (s)p99
12:07
DEAD END
'CPU is 48%, so it's not CPU'
The host dashboard shows 48% of ONE core, but the container is only ALLOWED half a core. As a share of what it can actually use, it's at 97%. The percentage was right; the denominator was wrong.
PromQL· Prometheus
```
rate(process_cpu_seconds_total{job="shoplite"}[1m])
```
result
{instance="shoplite:8080", job="shoplite"} 0.487
12:12
QUERY
Saturation, not utilisation: the event loop is waiting
Node runs JavaScript on one thread. When CPU is scarce, callbacks queue up waiting for their turn: EVENT-LOOP LAG. prom-client's default metrics include it. Lag at p99 of 800 ms means every request can wait almost a second before its code even starts.
PromQL· Prometheus
```
nodejs_eventloop_lag_p99_seconds{job="shoplite"}
```
result
{instance="shoplite:8080", job="shoplite"} 0.812
12:18
FINDING
Demand: 14 req/s × 50 ms CPU each = 0.7 cores. Supply: 0.5 cores
The container needs 140% of its allowance. The kernel's CFS quota throttles it for the rest of each 100 ms period, so work queues and latency grows until clients time out. Autoscaling on 'CPU > 70%' never fired, because the utilisation metric it watched was computed against the host.
12:44
RESOLVED
The expensive code path is reverted; CPU per request back to ~3 ms

Root cause

A code change made /products CPU-heavy. Demand exceeded the container's CPU LIMIT, so the process was throttled and requests queued. Monitoring showed CPU as a fraction of the host, which looked moderate, and nobody measured saturation (event-loop lag, throttling) or CPU against the limit.

02 The concepts behind it

USE: Utilisation, Saturation, Errors

For every resource (CPU, memory, disk, network, connection pools, threads), ask three questions. UTILISATION: how busy is it, as a fraction of capacity? SATURATION: how much work is WAITING for it (run queues, event-loop lag, pool wait time, throttled periods)? ERRORS: is it failing? Utilisation near 100% is fine if nothing waits; saturation is what users feel.

Utilisation needs the right denominator

rate(process_cpu_seconds_total[1m]) is CPU seconds per second, which is cores used. Divide by the LIMIT (0.5 here, ECS task CPU/1024, or the Kubernetes limit) to get real utilisation. With cAdvisor or Kubernetes metrics you also get container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total, the fraction of scheduling periods in which the container was throttled. Anything above a few percent means latency.

Runtime saturation signals

Every runtime has one: Node's event-loop lag, JVM GC pause time and thread-pool queue depth, Go's goroutine count and scheduler latency, Python's asyncio loop lag or worker-pool queue. They're often the earliest warning, rising before CPU graphs look worrying.

03 The fix

01Remove the expensive work and the scenario
The real-world fix is profiling (node --cpu-prof, flame graphs) to find the hot path. Scaling out only helps if the autoscaler is watching the right signal.
terminal
$ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"cpu-burn","enabled":false}'
── expected output ──
{"active":[]}

04 Make sure it never surprises you again

01Record CPU as a fraction of the limit
A recording rule stores the result of an expression as a new series, which is cheaper to query and gives the concept a name. The limit is a constant here; in Kubernetes, join against kube_pod_container_resource_limits.
obs-lab/prometheus/alerts.ymladd to fileyaml
```
  - name: shoplite-recording
    rules:
      - record: instance:cpu_utilisation_of_limit:ratio
        expr: rate(process_cpu_seconds_total{job="shoplite"}[1m]) / 0.5
```

02Alert on saturation, which is what users feel

obs-lab/prometheus/alerts.ymladd to fileyaml

      - alert: ShopLiteEventLoopSaturated
        expr: nodejs_eventloop_lag_p99_seconds{job="shoplite"} > 0.1
        for: 5m
        labels: { severity: warning, service: shoplite }
        annotations:
          summary: "Event-loop lag p99 is {{ $value | humanizeDuration }} — the process is CPU-starved or blocked"

05 Your turn: write the query

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.

With cpu-burn on, compute ShopLite's average CPU milliseconds per request.

Which default Node metric would tell you the process is spending lots of time in garbage collection?

06 Interview questions from this case

Explain the USE method and give a saturation metric for CPU.

A containerised service is slow but its CPU graph shows 50%. What might be going on?

0/4 · 0%

Command Palette

00 It starts

01 The investigation

p99 latency on /products jumps from 50 ms to ~900 ms

'CPU is 48%, so it's not CPU'

Saturation, not utilisation: the event loop is waiting

Demand: 14 req/s × 50 ms CPU each = 0.7 cores. Supply: 0.5 cores

The expensive code path is reverted; CPU per request back to ~3 ms