A budget table with p50 and p95
The milliseconds are simulated sample data, and the budgets are example values. Replace both with your own measurements.
A budget gives each stage of the query path a time limit, the way a trip has a time for each leg. With budgets you can see which stage is spending too much, decide what to cut first, and make time to first token feel fast even when the whole answer takes seconds.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write percentile(values, pct) using the nearest-rank rule and print the p50 and p95 of [12, 15, 11, 90, 14, 13, 12, 16, 15, 14].
Write over_budget(samples, budgets) returning the sorted names of the stages whose p95 is above their budget. Use the same percentile.
Explain it without notes
Why are budgets set at p95 or p99 and not at the average?
Why is the p95 of a request not the sum of its stages' p95 values?
What is time to first token and how does streaming change what users feel?
A RAG service is over its latency budget. How do you decide what to change first?
Expected output
stage budget p50 p95 status
embed 50 32 45 ok
search 80 49 200 OVER by 120
rerank 150 117 140 ok
first token 400 315 500 OVER by 100
whole request p95: 701 ms (budget 700)
sum of stage p95s: 885 ms
look at first: search