Blocking vs streaming: when the first words appear
Time is a simulated counter, so every language prints the same numbers: 400 ms of prefill, then 50 ms per token.
Streaming does not make the model faster. It shows the first words after the model starts writing instead of after it finishes, which is what the reader feels. It does nothing for programs that need the whole reply, such as JSON for code.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write streaming_gain(prefill_ms, ms_per_token, tokens) that returns how many times sooner the first token arrives than the whole reply, as an integer (round down).
Write should_stream(for_human, reply_tokens, must_check_first) that returns True only when a human reads the reply as it arrives, it has more than 30 tokens, and it does not need checking first.
Explain it without notes
What problem does streaming solve, and what does it not change?
What is the difference between time to first token and total time, and what drives each?
Give three situations where streaming does not help, and say why.
Why can a streaming reasoning model still feel slow to start?
Expected output
reply tokens blocking UI streaming UI sooner
short 4 600 ms 450 ms 1x
medium 40 2400 ms 450 ms 5x
long 200 10400 ms 450 ms 23x