cgroups: Limits, OOM Kills & Throttling

In one line

Control groups limit and account for CPU, memory, I/O, and process counts; container memory limits are enforced by the OOM killer, and CPU limits by throttling.

0/4 · 0%

Think of it like this

A shared office kitchen with rules: each team gets two burners (CPU) and one fridge shelf (memory). Overfill your shelf and your food gets thrown out (OOM kill); use more than two burners and you have to wait (throttling).

Key ideas

01
cgroups v2 (the default on modern Linux) organises processes in a tree under /sys/fs/cgroup, with controllers per resource: memory.max, cpu.max, io.max, pids.max. docker run --memory 512m --cpus 0.5 and Kubernetes limits write these files.
02
MEMORY: when a container exceeds memory.max, the kernel's OOM killer kills a process in it, usually the main one, and you see OOMKilled: true and exit code 137 (Observability course, Case 1.4). Page cache counts too, which can surprise file-heavy workloads.
03
CPU: cpu.max = 50000 100000 means 50 ms of CPU time per 100 ms period. Use more and the container is THROTTLED until the next period, which shows up as latency, not errors. cpu.stat reports nr_throttled and throttled_usec (Observability course, Case 1.3; SRE course, Shift 2.1).
04
Runtimes must know their limits: older JVMs sized heaps from HOST memory and got OOM-killed (Docker course, Topic 6.2); Node's heap default and Go's GOMAXPROCS have similar issues, so set them from the container's limits.

Code & diagrams

cgroups.shbash

Paths vary by distribution and cgroup driver (systemd vs cgroupfs); `docker inspect` and `systemd-cgls` help locate them.

docker run -d --name lim --memory 256m --cpus 0.5 nginx:alpine
CG=$(docker inspect -f '{{.Id}}' lim)
cat /sys/fs/cgroup/system.slice/docker-$CG.scope/memory.max    # 268435456
cat /sys/fs/cgroup/system.slice/docker-$CG.scope/cpu.max       # 50000 100000
cat /sys/fs/cgroup/system.slice/docker-$CG.scope/cpu.stat      # nr_throttled, throttled_usec
docker stats --no-stream lim
docker rm -f lim

Explain it without notes

A container exits with code 137. What does that tell you, and what do you check next?

Practice

Why can a container with a 1-CPU limit be slow even when the host has 32 idle cores?

Trade-offs

↔
Limits protect neighbours on shared hosts but cause OOM kills and throttling when set too tight; no limits risk noisy neighbours. Kubernetes separates requests (scheduling) from limits (enforcement) so you can choose per resource.

Done when you can

I can find a container's cgroup files and read its limits.
I can recognise OOM kills (137) and CPU throttling.
I size runtime heaps and threads from container limits.