Topic 6.3
Where Latency Comes From
In one line
A model call's time is network, queue, reading your prompt (prefill) and writing the answer one token at a time (decode). Knowing the four parts tells you what to shorten, and percentiles (p50, p95) tell you how slow your slowest users really are.
Think of it like this
Ordering at a restaurant. You walk there (network), wait for a table (queue), the cook reads your long order slip (prefill), then the dishes come out one by one (decode). A long order slip adds to the wait before the first dish. Many dishes add to the wait for the last one. Each part can be fixed in a different way.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Latency
- How long you wait, from sending a request to getting the result.
- Time to first token (TTFT)
- The wait before the first piece of the answer arrives: network + queue + prefill.
- Prefill
- The model reading all your input tokens before it writes anything.
- Decode
- The model writing the answer one token after another.
- Tokens per second
- How fast the model writes. Total decode time is output tokens divided by this.
- Percentile (p50, p95)
- p95 is the time that 95 percent of requests finish within. The slowest 5 percent take longer.
- Tail latency
- The slow end of the distribution (p95, p99). Users remember it.
Step by step
01Step 1: a stopwatch is not a diagnosis
"It takes 6 seconds" tells you nothing about what to fix. Start by splitting a call into the stages that happen in order.
02Step 2: a model of the stages
Prefill time is input tokens divided by the reading speed. Decode time is output tokens divided by the writing speed. Use whole milliseconds and integer division so every language prints the same. The speeds in the example are made-up round numbers.
def timing(input_tokens, output_tokens):
prefill = input_tokens * 1000 // PREFILL_TOK_PER_S
decode = output_tokens * 1000 // DECODE_TOK_PER_S
first = NETWORK_MS + QUEUE_MS + prefill
return first, first + decode03Step 3: two shapes of request
A document summary has a long prompt and a short answer: most of the time is prefill. A story has a short prompt and a long answer: most of the time is decode. The same model, two different fixes: trim the document for the first, ask for fewer words (or stream) for the second.
04Step 4: draw many requests, not one
Twenty simulated requests, made with the course's seeded generator, give a spread of times with an occasional slow one. Sort them and read off p50 and p95 using the nearest-rank method: the value at position ceil(p/100 × n).
05Step 5: compare mean with p95
Run the example: the mean sits between the typical time and the slow ones, and p95 is far above p50. The mean describes nobody's real experience. p95 tells you how bad it gets for one user in twenty.
06Step 6: choose the fix by stage
Prefill slow: shorter or cached prompts (Topic 6.6). Decode slow: fewer output tokens, a faster or smaller model, streaming so the user reads while it writes (Phase 5). Queue slow: another region, another provider, or a fallback (Topic 6.9). Network slow: servers closer to the provider, reused connections.
Try it yourself
- 1
Predict the two timings
Before running, compute the first-token time for 8,000 input tokens (40 + 60 + 8000 × 1000 / 5000). Then change the prompt to 4,000 tokens and predict the new total.
- 2
Halve the answer
Change the story's output from 600 to 300 tokens. By how many milliseconds does the total drop? Is it more or less than the drop from halving the long prompt?
- 3
Make the tail worse
Change the slow-request rule from
state % 10 == 0tostate % 5 == 0(in every language). What happens to the mean, p50 and p95?
Code & diagrams
The speeds are example numbers and the clock is simulated. The twenty request times come from the course's seeded generator.
Expected output
long prompt, short answer: first token 1700 ms, total 1900 ms (reading 1600 ms, writing 200 ms)
short prompt, long answer: first token 140 ms, total 6140 ms (reading 40 ms, writing 6000 ms)
mean 667 ms, p50 506 ms, p95 2410 ms, max 2560 msBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Judge by the mean
Make the slow requests much slower: change the penalty + 2000 to + 20000 in the loop (in every language). Run it and compare the mean with p50.
Myth vs fact
Myth
A faster model fixes all latency.
Fact
Network, queue and prefill don't change with output speed. Trim the prompt and the answer first, and measure which stage dominates.
Myth
The average response time is the number to track.
Fact
The mean hides the slow tail that real users hit. Track p50 and p95.
Myth
Latency is the provider's problem.
Fact
Prompt length, answer length, retries, serial calls and where your server runs are all yours.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
A request that calls the model three times in a row has three times the latency. Run independent calls concurrently (Topic 6.10), and don't chain calls a single prompt could handle.
- ▸
Retries and fallbacks add to tail latency: a timeout of 10 seconds plus a retry is 20 seconds at p99. Give each call a deadline and a total budget (Topic 6.9).
- ▸
Compare models by time to first token and tokens per second at your real prompt size, not by headline numbers. Speeds differ by provider, load and time of day.
- ▸
Streaming does not shorten total time. It improves how fast the answer feels (Topic 5.1). Choose it for reading, not for pipelines that wait for the whole reply.
- ▸
Queue time is invisible in your own code but shows up as a gap between a fast, short-prompt call and a slow one at busy hours. If time to first token varies a lot for identical prompts, suspect queueing or load on the provider, not your prompt, and compare p95 by hour of day before changing anything.
Remember this
- 1
Why this exists. Users judge a feature by how long they wait. "The model is slow" is not a diagnosis. If you don't know which part is slow, you will shorten the wrong thing. Splitting the time into parts tells you whether to trim the prompt, cap the answer, stream (Phase 5), change region, or choose a faster model.
- 2
The four parts. Network: the round trip to the provider. Queue: waiting for a free machine, which grows when the service is busy. Prefill: the model reads all input tokens before writing anything, quickly and in parallel, so long prompts add time before the first token. Decode: the model writes output tokens one after another, each depending on the previous, so time grows in proportion to the number of output tokens.
- 3
Two numbers to watch. Time to first token (TTFT) is network + queue + prefill: when the user first sees anything if you stream. Total time is TTFT plus decode. Output speed is measured in tokens per second. In the example, a long prompt with a short answer is slow to start but finishes soon, and a short prompt with a long answer starts at once but takes six seconds in total.
- 4
Shorter outputs are faster. Because decode is one token at a time, halving the answer length nearly halves the writing time. "Answer in two sentences", a lower
max_tokens, and structured output with short fields all make a feature faster, and cheaper (Topic 6.1). Reasoning models add hidden thinking tokens to decode time (Topic 1.8): the user waits for tokens they never see. - 5
Averages hide pain. If nine requests take 400 ms and one takes 2.4 seconds, the mean looks fine, but one user in ten has a bad time. Percentiles describe the spread: p50 (the median) is the time half the requests beat; p95 is the time 95 percent beat, so the slowest 5 percent are worse. Set goals and alerts on p95 or p99, not on the mean.
- 6
Measure it yourself. Record start time, first-token time and end time for every call, with the input and output token counts, and log them (Topic 11.4). The clock in these examples is simulated so the output is the same every run. In a real service you read a real clock around the call and send the numbers to your metrics system.
Explain it without notes
Name the four parts of a model call's latency and what shortens each.
Why does output length matter more for latency than input length?
What are p50 and p95, and why is p95 better for alerts than the mean?
A feature has a 4-second p95. How would you find which stage is responsible?
Practice
Write ttft_and_total(input_tokens, output_tokens) using 50 ms network+queue, 4,000 tokens per second prefill and 80 tokens per second decode. Return both in whole milliseconds. Print the result for 2,000 input and 160 output tokens.
Write percentile(values, p) using the nearest-rank method (sort, then take the value at position ceil(p/100 × n)) and print p50 and p90 of [120, 150, 130, 900, 140, 135, 160, 125, 145, 155].
Trade-offs
- ↔
Shorter answers are faster and cheaper but may be less helpful; measure quality when you cut.
- ↔
Streaming improves perceived speed but adds complexity and doesn't help when the whole reply is needed before acting.
- ↔
Percentiles need enough samples: p99 of ten requests is the maximum. Collect more data before alerting on high percentiles.
Done when you can
Done when you can name the four stages of a model call's latency.
Done when you can explain time to first token versus total time.
Done when you can compute p50 and p95 from a list of times.
Done when you can say which stage to fix for a slow long-prompt call and for a slow long-answer call.