Unit 5.3 · Performance Engineering
Finding Bottlenecks and Planning Capacity
A repeatable bottleneck workflow, USE and RED methods, Little's Law and the utilisation knee, profilers and flame graphs, the usual suspects in each tier, and capacity planning with headroom.
Start here
The mental model
A system is a chain of pipes of different widths: load balancer, app pods, connection pool, database, disk. Throughput is limited by the NARROWEST pipe, and making any other pipe wider changes nothing. Performance work is finding that narrow pipe, widening it, and then finding the next one, because there always is a next one.
The most useful fact in the whole field: as a resource approaches 100% busy, waiting time doesn't grow gently, it explodes. That's why a database at 60% CPU is fine, at 85% is getting slow, and at 95% is an outage.
Go deeper
How it works inside
01The workflow
1) Define the goal in numbers (e.g. 400 req/s at p95 < 300 ms, < 0.5% errors). 2) Reproduce with a load test while recording metrics for every tier. 3) Find the FIRST resource to saturate as load rises: check utilisation, saturation (queues), and errors. 4) Form a hypothesis and change ONE thing. 5) Re-run the same test and compare. 6) Stop when the goal is met with headroom. Changing five things at once, or tuning without measuring, is how teams spend weeks making no progress.
02USE and RED
USE (Brendan Gregg) for every RESOURCE (CPU, memory, disk, network, connection pool, thread pool, Kafka partition): Utilisation (% busy), Saturation (queued work: run queue, iostat await, pool wait count, consumer lag), Errors. RED (Tom Wilkie) for every SERVICE: Rate, Errors, Duration. RED tells you WHICH service is slow; USE tells you WHICH RESOURCE inside it is the cause (Observability course).
03Little's Law and the utilisation knee
LITTLE'S LAW: L = λ × W. Items in the system = arrival rate × time each spends in it. At 200 req/s with 250 ms average latency, 50 requests are in flight on average, so a pool of 20 DB connections with 100 ms queries (200 × 0.1 = 20 busy) is already at its limit. It sizes pools, threads, and consumers from numbers you already have.
QUEUEING: for a simple single-server queue, the average wait grows like U / (1 − U). At 50% utilisation the wait is about the service time; at 80%, 4×; at 90%, 9×; at 95%, 19×. Hence 'keep headroom': plan for peak utilisation of roughly 60–70% on critical resources.
04Profilers and flame graphs
When CPU is the bottleneck, a PROFILER shows which code uses it. Sampling profilers capture stack traces many times per second with low overhead, safe in production: async-profiler or Java Flight Recorder for the JVM, pprof for Go, py-spy for Python, perf plus eBPF tools for anything on Linux. A FLAME GRAPH displays them: width = share of samples, so the widest plateaus near the top are where the time goes. Continuous profiling (Pyroscope, Grafana Cloud Profiles, Parca) keeps profiles over time so you can compare before and after a deploy.
05The usual suspects, by tier
APP: CPU limits causing throttling (Observability course, CPU throttling), GC pauses and undersized heap, thread or connection pools too small (requests queue inside the app while CPU looks idle), N+1 queries, synchronous calls to slow dependencies without timeouts. DATABASE: missing indexes, lock contention, connection limits, IOPS limits (Part 1). CACHE: hit ratio dropping, a hot key, one slow command (Part 3). MESSAGING: consumer lag from too few partitions or slow processing (Part 2). NETWORK: cross-AZ hops, TLS handshakes without connection reuse, NAT gateway port exhaustion, and DNS lookups on every request (Networking course).
06Capacity planning
From load tests you know each component's limit (one pod handles ~120 req/s at the p95 target; the DB handles ~1,800 queries/s at 65% CPU). From traffic history and business plans you forecast peak demand (last year's sale peak × growth × a safety factor). Capacity = forecast peak ÷ per-unit limit, plus headroom for losing an AZ (N+1 or N+2) and deploys. Check autoscaling can react fast enough for spikes (pre-scale before a known event), and that downstream limits (DB connections, third-party rate limits, quotas) scale too, not just pods.
Do it
Hands-on lab
- 1
USE check on a Linux host under load
While a load test runs, these commands answer utilisation and saturation for CPU, memory, disk, and network (Linux course).
terminal$ vmstat 1 3 # r = run queue (saturation), us/sy/wa = CPU utilisationiostat -xz 1 2 # %util and r_await/w_await per devicesar -n DEV,TCP 1 1 # network throughput, retransmits── expected output ──procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st14 0 0 412180 51240 2911004 0 0 4 1820 9512 18231 81 12 6 1 0Device r/s w/s rkB/s wkB/s r_await w_await aqu-sz %utilnvme1n1 2.0 2890.0 16.0 46240.0 0.41 3.92 11.3 98.7 - 2
Apply Little's Law to a connection pool
The API does 3 queries per request at ~8 ms each (24 ms of DB time per request) and must handle 600 req/s. Busy connections ≈ 600 × 0.024 = 14.4 on average. With bursts and the utilisation knee in mind, a pool of ~24 across all pods (behind PgBouncer) is reasonable; 10 would queue constantly; 200 would just add contention at the database.
terminal$ python3 -c "rate=600; db_time=3*0.008; busy=rate*db_time; print(f'busy connections ≈ {busy:.1f}; at 60% target utilisation → pool ≈ {busy/0.6:.0f}')"── expected output ──busy connections ≈ 14.4; at 60% target utilisation → pool ≈ 24 - 3
Profile a hot JVM service with async-profiler
Attach to the running JVM for 30 seconds during the load test and produce an HTML flame graph. Look for the widest frames.
terminal$ kubectl exec deploy/api -- /opt/async-profiler/bin/asprof -d 30 -e cpu -f /tmp/cpu.html 1kubectl cp $(kubectl get pod -l app=api -o name | head -1 | cut -d/ -f2):/tmp/cpu.html ./cpu.html── expected output ──Profiling for 30 secondsDone - 4
Profile a Go service with pprof
Go services that import
net/http/pprofexpose profiles over HTTP.-httpopens an interactive flame graph.terminal$ kubectl port-forward deploy/cart 6060:6060 &go tool pprof -top -seconds 30 http://localhost:6060/debug/pprof/profile | head -6── expected output ──Showing nodes accounting for 2.61s, 87.00% of 3s totalflat flat% sum% cum cum%1.12s 37.33% 37.33% 1.12s 37.33% encoding/json.(*decodeState).object0.48s 16.00% 53.33% 0.48s 16.00% runtime.mallocgc0.31s 10.33% 63.67% 2.02s 67.33% main.(*CartService).Price
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| Target peak utilisation | — | How busy critical resources may get at forecast peak. | ~60–70% for latency-sensitive tiers; lower for single points like a primary DB. |
| App thread / worker pools | framework-specific | Concurrent requests a pod processes. | Size with Little's Law from request rate and latency; too small queues internally with idle CPU. |
| HPA target CPU | — | Autoscaling trigger. | Around 60–70% so scaling starts before the knee; scale on RPS or latency if CPU isn't the constraint. |
| Timeouts & retries | often infinite | How long to wait on dependencies. | Set on every call, below the caller's own timeout, with retry budgets to avoid retry storms. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
Adding pods made it slower
Checkout p95 is 900 ms at 400 req/s. The team scales the API from 10 to 30 pods. p95 gets WORSE (1.4 s) and DB CPU hits 98%.
Drill #2
Low CPU, high latency
Under load, the API's p99 is 2 s but its CPU is only 30% and the database is idle.
The bigger picture
Connects to
System Design · Capacity Estimation
QPS/RPS, storage, bandwidth, memory — the arithmetic that makes every HLD answer defensible.
System Design · Numbers Every Engineer Should Know
Design decisions are only as good as the numbers behind them.
Observability · CPU throttling
The container-limits bottleneck that looks like 'low CPU, high latency'.
Linux · Logs and monitoring
top, vmstat, iostat: the USE toolkit on a single host.
SRE · Traffic surge
Capacity limits met in production, live.
Kubernetes · Requests, limits & QoS
How CPU and memory limits change performance under load.
Prove it
Interview questions
An API is slow under load. Walk through how you find the bottleneck.
Explain Little's Law and give a practical use.
Why keep headroom instead of running at 90% utilisation?