Topic 5.1
Why Stream
In one line
Streaming does not make the model faster. It shows the first words after the model starts writing instead of after it finishes, which is what the reader feels. It does nothing for programs that need the whole reply, such as JSON for code.
Think of it like this
A restaurant that brings the bread basket right away, then the soup, then the main course, compared with one that makes you wait forty minutes and then puts everything on the table at once. The kitchen needs the same total time in both. In the first, you are eating after two minutes.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Token
- A small piece of text, often a word or part of a word. Models read and write tokens, and providers bill per token.
- Streaming
- Sending the reply in many small pieces as it is produced, instead of one message when it is finished.
- Time to first token (TTFT)
- The wait from sending the request until the first piece of the reply arrives.
- Prefill
- The model reading your whole prompt before it writes anything. A longer prompt means a longer prefill.
- Perceived latency
- How long it feels like waiting. Showing something early lowers it even if the total time stays the same.
- Reasoning tokens
- Hidden thinking tokens some models write before the visible answer. They take time and are billed as output.
- Spinner
- An animation that tells the user to wait. Streaming replaces a long spinner with text appearing.
Step by step
01Step 1: the blocking version
The simplest call sends the request and waits for the whole reply. The user sees nothing for the full generation time. For a 4-token answer that is 600 ms and nobody minds. For a 200-token answer it is over 10 seconds.
02Step 2: split the wait into two numbers
Write the time as two parts: a wait before the first token (here just the prefill, 400 ms, ignoring network time) and then 50 ms per token (an example speed). The first example computes both the blocking time (prefill + tokens * 50) and the streaming time to first token (prefill + 50) for short, medium and long replies, and divides one by the other.
PREFILL_MS = 400
MS_PER_TOKEN = 50
tokens = 200
print(PREFILL_MS + MS_PER_TOKEN * tokens) # 10400: blocking shows text now
print(PREFILL_MS + MS_PER_TOKEN) # 450: streaming shows text now03Step 3: notice who gains
The gain grows with reply length: about the same for 4 tokens, 5 times sooner for 40, 23 times sooner for 200. Streaming matters most where answers are long: chat, summaries, explanations, code generation.
04Step 4: find the reply you cannot stream usefully
The second example feeds a JSON reply to a parser piece by piece. It reports not valid yet four times and valid JSON only after the closing brace. A program that needs temp gets nothing from the early pieces.
05Step 5: measure in the real world
Run the lab with streaming on. The lab shows time to first token and total time for the same model. Compare a one-word answer with a 150-word answer: TTFT stays nearly the same, total time grows. Then try a reasoning model and notice the hidden thinking before the first word.
06Step 6: decide per feature
Stream when a human reads the text as it arrives and the reply is more than a sentence or two. Do not stream when a program consumes the reply, or when you must validate it first. Many products mix both: stream the chat, do not stream the classification call behind it.
Try it yourself
- 1
Predict, then run
In the first example, change
MS_PER_TOKENto 20. Predict the three "sooner" numbers (total divided by first, rounded down) before you run it. What happens to the gain for the short reply? - 2
A slower prefill
Set
PREFILL_MSto 3000, as for a very long prompt. Run again. Streaming still helps, but by how much for the long reply? Long prompts shrink the benefit because prefill cannot be streamed. - 3
Find the first valid piece
In the JSON example, add one more field (
,"unit":"C") as extra pieces. On which piece does the text first become valid JSON? Why can it never be earlier than the last closing brace? - 4
Try a reasoning model
In the lab, switch the model to
openai/gpt-oss-20b(set max tokens to at least 500) and send again. Compare the time to first token with the other model, then set reasoning effort to low and compare again. Hidden thinking tokens come before the first visible word.
Code & diagrams
Time is a simulated counter, so every language prints the same numbers: 400 ms of prefill, then 50 ms per token.
Expected output
reply tokens blocking UI streaming UI sooner
short 4 600 ms 450 ms 1x
medium 40 2400 ms 450 ms 5x
long 200 10400 ms 450 ms 23xStreaming a reply that a program must parse gains nothing: it only becomes valid at the end.
Expected output
after piece 1: {"city" not valid yet
after piece 2: {"city":"Pune" not valid yet
after piece 3: {"city":"Pune","temp" not valid yet
after piece 4: {"city":"Pune","temp":31 not valid yet
after piece 5: {"city":"Pune","temp":31} valid JSON
first usable JSON: piece 5 of 5Try: Press Send with streaming on and read the time to first token and the total time. Then change the request to ask for a single word and send again. Which number changed, and which stayed nearly the same?
Goals
- The reply finished on its own
- The reply is 220 words or fewer
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Stream the thing a program needs
In the JSON example, change the last piece from "}" to "}}" (an extra closing brace, as a broken model reply might send).
Myth vs fact
Myth
Streaming makes the model answer faster.
Fact
The model writes at the same speed and the total time is the same or slightly longer. Streaming only shows the words earlier.
Myth
Streaming costs extra money.
Fact
Providers bill the same tokens either way. The cost is engineering effort: partial states, cancelling, broken connections and proxies.
Myth
Always stream everything.
Fact
Stream what a person reads as it arrives. Do not stream replies that a program parses, tool-call arguments you must run whole, or answers you must check first.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Track three numbers per request: TTFT, tokens per second after the first token, and total time. They have different causes: TTFT grows with prompt length and provider queueing, speed depends on the model and load. A dashboard with only total time hides which one regressed (Topic 6.3, Topic 11.4).
- ▸
A long, repeated prompt raises TTFT on every call. Provider prompt caching (Topic 6.6) can cut prefill time for a stable prefix, so putting fixed text first helps both cost and TTFT.
- ▸
Reasoning models change what "first token" means. Measure time to the first visible content token, and show an honest status while hidden thinking runs. Lower reasoning effort trades some quality for a faster start.
- ▸
Streaming holds a connection open for the whole generation. At scale, that is many open connections per server: budget file descriptors and idle timeouts, and make sure load balancers do not cut long requests (Topic 5.4).
Remember this
- 1
Why this exists. A model produces its answer one token (a small piece of text: a word, part of a word or a punctuation mark) at a time. A 200-token answer at 50 milliseconds per token takes 10 seconds of writing. If your app waits for the complete reply and then shows it, the user stares at a spinner for 10 seconds. If your app shows each token as it arrives, the first words appear after well under a second, and the user reads while the model writes.
- 2
Two numbers matter, not one. Time to first token (TTFT) is the wait before anything appears: network, queueing at the provider, and prefill (the model reading your whole prompt before it writes anything). Generation speed is how many tokens per second come after that. Total time is roughly
TTFT + tokens / speed. A long prompt raises TTFT, a long answer raises total time. Streaming attacks the part the user feels most: the blank screen. - 3
Reading speed makes it work. People read a few words per second. Models generate tokens faster than that, so the reader is rarely waiting for the next word once the first has appeared. The total time is the same (streaming even adds a little overhead for the extra framing), but the perceived latency drops from the whole reply to the first token. The example below counts it: with a 400 ms prefill and 50 ms per token, a 200-token reply is ready after 10,400 ms but its first token appears at 450 ms.
- 4
When streaming does not help. If the next step needs the whole reply, you gain nothing by receiving it in pieces. Four cases: JSON that a program will parse (the second example shows it stays invalid until the last brace), a classification label of one word, a reply that you must check for policy violations before any user sees it, and tool-call arguments, which you can only run when complete. Short replies finish so fast that streaming adds code and no benefit. Batch jobs that nobody watches should not stream. Topic 5.6 shows a middle path for structured output.
- 5
Reasoning models move the wait. A reasoning model spends hidden reasoning tokens thinking before it writes the first visible word. Those tokens take time and cost money, but nothing visible streams while they are produced, so TTFT can be several seconds even though streaming is on. Try it in the lab below with the model switched to a reasoning model. For these models, show a status such as "Thinking..." until the first content token arrives.
- 6
What streaming costs you in engineering. Each reply is now a sequence of events with a start, a middle and an end, instead of one value. You must handle a reply that is half done, the user leaving, a connection that drops, proxies that hold bytes back, and metrics for TTFT and gaps between tokens. The rest of this phase builds those pieces, one by one. Tokens cost the same whether streamed or not.
Explain it without notes
What problem does streaming solve, and what does it not change?
What is the difference between time to first token and total time, and what drives each?
Give three situations where streaming does not help, and say why.
Why can a streaming reasoning model still feel slow to start?
Practice
Write streaming_gain(prefill_ms, ms_per_token, tokens) that returns how many times sooner the first token arrives than the whole reply, as an integer (round down).
Write should_stream(for_human, reply_tokens, must_check_first) that returns True only when a human reads the reply as it arrives, it has more than 30 tokens, and it does not need checking first.
Trade-offs
- ↔
Streaming lowers perceived latency but adds partial states, cancellation and broken-connection handling to every client.
- ↔
Checking a reply before showing it protects users but means no streaming (or streaming only after the check passes).
- ↔
Smaller models and shorter prompts lower both TTFT and total time, and need no streaming code at all; streaming and a faster model are different levers.
Done when you can
Done when you can explain time to first token and total time, and what drives each.
Done when you can compute, from a prefill time and a speed, when the first token and the last token arrive.
Done when you can list the replies that should not be streamed and say why.
Done when you can read time to first token for a real model in the lab.