Stages of a call, and p50 versus p95
The speeds are example numbers and the clock is simulated. The twenty request times come from the course's seeded generator.
A model call's time is network, queue, reading your prompt (prefill) and writing the answer one token at a time (decode). Knowing the four parts tells you what to shorten, and percentiles (p50, p95) tell you how slow your slowest users really are.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write ttft_and_total(input_tokens, output_tokens) using 50 ms network+queue, 4,000 tokens per second prefill and 80 tokens per second decode. Return both in whole milliseconds. Print the result for 2,000 input and 160 output tokens.
Write percentile(values, p) using the nearest-rank method (sort, then take the value at position ceil(p/100 × n)) and print p50 and p90 of [120, 150, 130, 900, 140, 135, 160, 125, 145, 155].
Explain it without notes
Name the four parts of a model call's latency and what shortens each.
Why does output length matter more for latency than input length?
What are p50 and p95, and why is p95 better for alerts than the mean?
A feature has a 4-second p95. How would you find which stage is responsible?
Expected output
long prompt, short answer: first token 1700 ms, total 1900 ms (reading 1600 ms, writing 200 ms)
short prompt, long answer: first token 140 ms, total 6140 ms (reading 40 ms, writing 6000 ms)
mean 667 ms, p50 506 ms, p95 2410 ms, max 2560 ms