Unit 5.1 · Performance Engineering
Load Testing Fundamentals with k6
The types of performance test, the numbers that matter (throughput, percentiles, errors), open vs closed workload models, coordinated omission, and a k6 test with thresholds that gates CI.
Start here
The mental model
A load test is a rehearsal: you send fake customers at the system in a controlled way and watch what bends and what breaks. Each TYPE of test asks a different question. Can it handle normal peak traffic (LOAD)? Where does it break (STRESS/BREAKPOINT)? Can it survive a sudden rush (SPIKE)? Does it degrade over hours, from leaks or filling disks (SOAK)? Does the script even work (SMOKE)?
The answer is never one number. It's a curve: as load rises, latency stays flat, then bends upward sharply near saturation. Your job is to find where the knee is and keep production comfortably to the left of it.
Go deeper
How it works inside
01The numbers that matter
THROUGHPUT: requests (or transactions) per second completed. LATENCY as PERCENTILES: p50 is the typical user, p95/p99 are the unlucky ones, and averages hide them (Observability course, averages lie). ERROR RATE: a fast error is not a success. SATURATION of resources during the test: CPU, memory, connections, queue depth, GC. A result without the matching resource graphs tells you THAT it's slow, not WHY.
02Open vs closed models
A CLOSED model has a fixed number of virtual users (VUs), each sending a request, waiting for the response, then sending the next. When the system slows down, users send less, so the load politely drops, which real internet traffic never does. An OPEN model sends requests at a fixed ARRIVAL RATE (e.g. 200 per second) regardless of how slow responses are, like real users arriving independently. Use open models (k6 constant-arrival-rate / ramping-arrival-rate) to find true limits of public-facing services.
COORDINATED OMISSION: in a closed model, while the system is stalled, the tool isn't sending requests, so it doesn't record the requests that WOULD have been waiting. The measured p99 looks far better than reality. Arrival-rate executors avoid most of it.
03Realistic tests
Model real user journeys (browse → search → add to cart → checkout) with realistic proportions and think time, use production-like data volumes (a test against an empty database proves nothing about indexes), randomise inputs so caches aren't artificially perfect, and test from outside the cluster through the real load balancer. Run against a production-like environment, never shared staging during someone else's demo, and tell the on-call engineers before you start.
04Performance tests in CI
A small smoke-load test on every merge with THRESHOLDS (p(95) < 300ms, errors < 1%) catches regressions early: the pipeline fails if a change makes checkout 40% slower. Bigger stress and soak tests run nightly or before launches. Canary analysis (GitOps course, Mission 2.2) is the production version of the same idea.
Do it
Hands-on lab
- 1
A target to test
Any HTTP service works. Grafana's QuickPizza demo app has a fast home page and a heavier recommendation API so you can see the difference.
terminal$ docker run -d --name target -p 3333:3333 ghcr.io/grafana/quickpizza-local:latestcurl -s -o /dev/null -w '%{http_code} %{time_total}s\n' localhost:3333/── expected output ──200 0.004s - 2
Write a k6 test with stages and thresholds
k6 scripts are JavaScript, but the engine is Go (no Node.js), and one machine can generate thousands of requests per second. The
ramping-arrival-rateexecutor is an OPEN model. Thresholds make the run fail (non-zero exit code) when SLO-like targets are missed.loadtest.jswhole filejavascript import http from "k6/http"; import { check, sleep } from "k6"; export const options = { scenarios: { shoppers: { executor: "ramping-arrival-rate", startRate: 10, timeUnit: "1s", preAllocatedVUs: 50, maxVUs: 500, stages: [ { target: 50, duration: "1m" }, // warm up { target: 200, duration: "3m" }, // expected peak { target: 400, duration: "2m" }, // 2x peak: find the knee { target: 0, duration: "30s" }, ], }, }, thresholds: { http_req_failed: ["rate<0.01"], // < 1% errors http_req_duration: ["p(95)<300", "p(99)<800"], // ms "http_req_duration{name:recommend}": ["p(95)<500"], }, }; const BASE = __ENV.BASE_URL || "http://localhost:3333"; export default function () { const home = http.get(`${BASE}/`, { tags: { name: "home" } }); check(home, { "home 200": (r) => r.status === 200 }); const rec = http.post(`${BASE}/api/pizza`, JSON.stringify({ maxCaloriesPerSlice: 1000 }), { headers: { "Content-Type": "application/json", Authorization: "token abcdef0123456789" }, tags: { name: "recommend" }, }); check(rec, { "recommend 200": (r) => r.status === 200 }); sleep(Math.random() * 2); // think time } - 3
Run it and read the summary
Look at p95/p99 per tagged request, the failure rate, and
dropped_iterations: if k6 couldn't start iterations at the requested rate (not enough VUs), the system is already saturated.terminal$ k6 run loadtest.js── expected output ──█ THRESHOLDShttp_req_duration✓ 'p(95)<300' p(95)=212.4ms✗ 'p(99)<800' p(99)=1.31shttp_req_duration{name:recommend}✗ 'p(95)<500' p(95)=742.8mshttp_req_failed✓ 'rate<0.01' rate=0.42%█ TOTAL RESULTSchecks.........................: 99.58% 118412 out of 118912http_reqs......................: 118912 297.4/sdropped_iterations.............: 1843 4.6/sERRO[0391] thresholds on metrics 'http_req_duration, http_req_duration{name:recommend}' have been crossed - 4
Stream results to Prometheus and Grafana
Correlate load with resource graphs by sending k6 metrics to Prometheus (remote write) and viewing them next to your service dashboards (Observability course).
terminal$ K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write k6 run -o experimental-prometheus-rw loadtest.js - 5
Gate CI on the thresholds
The job fails when thresholds fail. Keep CI runs short (1–2 minutes at modest load) and run big tests on a schedule.
.github/workflows/perf.ymlwhole fileyaml name: perf-smoke on: [pull_request] jobs: k6: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: grafana/setup-k6-action@v1 - uses: grafana/run-k6-action@v1 env: BASE_URL: ${{ vars.PREVIEW_URL }} with: path: tests/perf/smoke.js
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| executor | per-vu-iterations | How k6 generates load. | ramping-arrival-rate / constant-arrival-rate (open model) for services; VU-based for session-like workloads. |
| thresholds | none | Pass/fail criteria. | Derive from SLOs (SRE course): p95/p99 latency and error rate per important endpoint. |
| preAllocatedVUs / maxVUs | — | VU pool for arrival-rate executors. | Enough that dropped_iterations stays 0 until the system itself saturates. |
| Load generator location | wherever you run it | Where traffic comes from. | Outside the cluster, close to users' path; never on the same node as the target. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
Great results, terrible launch
The team load-tested checkout at 1,000 VUs: p99 120 ms. At launch, real traffic of ~300 req/s saw p99 of 4 s and timeouts.
Drill #2
The load generator was the bottleneck
A stress test plateaus at exactly 2,100 req/s whatever the target, and server CPU is only 35%.
Decide
Types of performance test
| Test | Question it answers | Shape | Typical length |
|---|---|---|---|
| Smoke | Does the script and system work at all? | Tiny constant load | 1 min |
| Load | Does it meet targets at expected peak? | Ramp to peak, hold | 15–60 min |
| Stress / breakpoint | Where and how does it break? | Keep ramping until failure | Until it breaks |
| Spike | Can it absorb a sudden rush (sale, push notification)? | Jump from low to very high | Minutes |
| Soak / endurance | Does it degrade over time (leaks, disk, connections)? | Moderate constant load | 4–24 h |
The bigger picture
Connects to
System Design · Capacity Estimation
QPS/RPS, storage, bandwidth, memory — the arithmetic that makes every HLD answer defensible.
Observability · Averages lie
Why percentiles, not averages, are the result that matters.
SRE · Negotiating SLOs
Thresholds should come from SLOs.
SRE · Launch readiness
Load tests are a launch gate.
CI/CD · Build and test stages
Where a perf smoke test fits in the pipeline.
Prove it
Interview questions
What's the difference between load, stress, spike, and soak testing?
What is coordinated omission?
How do you make load tests realistic?