Topic 11.4
Latency Budgets
In one line
A budget gives each stage of the query path a time limit, the way a trip has a time for each leg. With budgets you can see which stage is spending too much, decide what to cut first, and make time to first token feel fast even when the whole answer takes seconds.
Think of it like this
You must reach the airport in 60 minutes. You allow 10 to get out of the house, 25 for the road, 15 for the car park and 10 for security. When you are late, you do not shout at the whole trip. You look at which leg overspent. A latency budget is that plan for a question: embed 50 ms, search 80 ms, re-rank 150 ms and so on.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Latency
- How long a request takes, from the user asking to the answer arriving.
- Budget
- The time a stage is allowed to use.
- p50 / p95 / p99
- The time that 50, 95 or 99 percent of requests are faster than.
- Nearest rank
- A way to pick a percentile: sort, then take the value at position ceil(p/100 x n).
- Time to first token (TTFT)
- The wait until the first word of a streamed answer appears.
- Deadline
- The moment by which a request must be finished, passed down to every stage.
- Tail latency
- The slow end of the distribution (p95, p99), where users' bad experiences live.
Step by step
01Step 1: split the path into stages
Embed the question, search, re-rank, build the prompt, wait for the first token. Give each a name and time it separately (traces, Topic 11.6, make this free). Small stages like building the prompt can share one "other" line.
02Step 2: collect samples
Record the time of every stage for many requests. The example uses ten per stage for readability; use at least a few hundred. Keep real traffic in mind: test questions are shorter than real ones.
03Step 3: compute p50 and p95 per stage
Sort and index. percentile(values, 95) returns the value at rank ceil(0.95 n). Print a table of budget, p50 and p95 and mark the rows over budget.
def percentile(values, pct):
ordered = sorted(values)
rank = (pct * len(ordered) + 99) // 100 # ceil(pct * n / 100) with whole numbers
return ordered[rank - 1]04Step 4: read the table
In the example search (p95 200 ms against 80) and first token (500 ms against 400) are over; embed and re-rank are fine. Search is furthest over its budget, so it is first on the list. The typical request (p50) is fine everywhere: this is a tail problem, which points to a slow shard or a cold cache, not a slow algorithm.
05Step 5: apply the cut-first order
Cache, then parallelise, then make cheaper, then drop under pressure. Change one thing, re-measure, and keep a note of what each change did to both latency and answer quality.
06Step 6: add a deadline
Pass deadline - now to each stage. If less than the re-ranker's p95 remains, skip the re-ranker and answer from search order. You trade a little quality on slow requests for not missing the deadline on any.
Try it yourself
- 1
Fix the tail
Change the 200 in the search samples to 70 and run. Which stage is looked at first now? What does the whole-request p95 do?
- 2
Add a stage
Add a stage "prompt" with samples around 15 ms and a budget of 30. Does the sum of p95s still differ from the whole-request p95? Why?
- 3
Use more samples
Duplicate each list so there are 20 values (the outlier now appears twice). Compute the p95 by hand: which rank is it? Does the answer change?
Code & diagrams
The milliseconds are simulated sample data, and the budgets are example values. Replace both with your own measurements.
Expected output
stage budget p50 p95 status
embed 50 32 45 ok
search 80 49 200 OVER by 120
rerank 150 117 140 ok
first token 400 315 500 OVER by 100
whole request p95: 701 ms (budget 700)
sum of stage p95s: 885 ms
look at first: searchBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Round the rank down
In the Python tab, change rank = (pct * len(ordered) + 99) // 100 to rank = (pct * len(ordered)) // 100 and run.
Myth vs fact
Myth
Average latency is a good enough number.
Fact
Averages hide the slow requests that users complain about. A service with a 300 ms average and a 4 s p99 feels broken to one user in a hundred, and that user is somebody.
Myth
The model call is the only part that matters.
Fact
It is usually the largest, but search tails, re-ranking and cold caches often decide whether you meet the deadline. Measure all stages.
Myth
Streaming makes the system faster.
Fact
It does not shorten the total time. It shortens the time to first token, which is what the user feels first.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Stage budgets should be set from a user target backwards: if p95 time to first token must be under 1.2 s, subtract the model's own start-up (measure it) and divide what remains among the earlier stages.
- ▸
Parallelism is free speed: the permission lookup, the query embedding and a keyword search have no dependence on each other and can start together. The path's time becomes the slowest branch, not the sum.
- ▸
Watch cold starts: the first request after a deploy or scale-up pays for loading the index into memory or warming a model. Warm instances before sending traffic (readiness probes), and keep a few always on for latency-critical paths.
- ▸
A re-ranker (Phase 6) scores each candidate with a model, so its time grows with the number of candidates. Halving the candidates is often the cheapest big win on this path; measure the quality change before accepting it.
- ▸
The Observability course (
/observability) covers latency histograms and SLOs; this topic uses them for one pipeline. LLM Engineering Topic 6.9 covers deadlines and circuit breakers for model calls.
Remember this
- 1
Why budgets exist. "The system feels slow" cannot be fixed, because it does not say where. A budget turns it into a table: stage, allowed time, measured time. Each owner knows their number, and a regression shows up as one red row instead of a vague complaint. It also stops the quiet creep where every stage adds "just 100 ms".
- 2
Measure percentiles, not averages. The p50 (median) is what a typical request sees. The p95 is the time that 95 out of 100 requests beat; one in twenty is slower. Users remember the slow ones, so budgets are set at p95 (or p99). The example uses the nearest rank rule: sort the values and take the one at position
ceil(p/100 × n). With only ten samples p95 is simply the slowest, which is why you need hundreds of requests in practice. - 3
Percentiles do not add. The sum of every stage's p95 (885 ms in the example) is not the p95 of the whole request (701 ms): the unlucky slow requests rarely hit every stage at once. So give the whole request its own budget and check it directly, and treat the stage budgets as a guide to where to look.
- 4
Time to first token (TTFT). A language model can start streaming its answer before it has finished. The user feels the delay until the first word appears, not the time to the last word. For RAG, TTFT is everything before generation plus the model's own start-up time. Streaming (LLM Engineering Phase 5) lets a 4-second answer feel like a 1-second one, provided the stages before it stay inside budget.
- 5
What to cut first. Look at the stage furthest over its budget, then ask in this order. (1) Can it be skipped by a cache (Topic 11.3)? (2) Can it run in parallel with another stage, like embedding the question while loading the user's permissions? (3) Can it be made cheaper: smaller k, fewer re-ranked candidates, a smaller model? (4) Can it be dropped under pressure, such as skipping re-ranking when search was slow? Quality changes need measuring (Phase 9); a cache costs no quality at all, so try it first.
- 6
Timeouts and deadlines. Give the whole request a deadline and pass the remaining time to each stage. A stage that has no time left should return what it has or skip itself, not wait. Without it, one slow dependency makes every request wait, queues grow and the whole service slows down. The Build topic has a request that skips re-ranking when search ran long.
Explain it without notes
Why are budgets set at p95 or p99 and not at the average?
Why is the p95 of a request not the sum of its stages' p95 values?
What is time to first token and how does streaming change what users feel?
A RAG service is over its latency budget. How do you decide what to change first?
Practice
Write percentile(values, pct) using the nearest-rank rule and print the p50 and p95 of [12, 15, 11, 90, 14, 13, 12, 16, 15, 14].
Write over_budget(samples, budgets) returning the sorted names of the stages whose p95 is above their budget. Use the same percentile.
Trade-offs
- ↔
Tight budgets: a fast service, with pressure to cut quality-giving stages like re-ranking.
- ↔
Skipping stages under load: protects the deadline, but answer quality varies by moment and must be logged.
- ↔
Streaming: better perceived speed, harder to apply output checks before the user sees text (Phase 8).
Done when you can
Done when you can write down a budget per stage and a total for a RAG request.
Done when you can compute a percentile by the nearest-rank rule.
Done when you can explain why per-stage p95 values do not add up.
Done when you can explain time to first token and what streaming changes.
Done when you can order the options for cutting latency: cache, parallelise, cheapen, drop.