PHASE 2Beginner ~25 min· topic 1 of 8

Topic 2.1

Why Streaming

In one line

Streaming does not make a model finish sooner. It lets the user start reading after the first few tokens instead of staring at a blank box until the last one. It costs more code and creates partial results to handle, and it does not suit short answers or JSON you cannot show half-finished.

Think of it like this

Ordering a four-course meal at a restaurant. In one restaurant the waiter brings everything together when the last dish is ready: you sit at an empty table for twenty minutes, then eat. In the other, each dish is brought as soon as it is cooked: you are eating after three minutes. The kitchen takes exactly the same time in both. Only the waiting is different. Streaming is the second restaurant.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Time to first token (TTFT): How long the user waits before the first piece of the answer appears.
Total time: How long until the whole answer has arrived. Streaming does not change this much.
Perceived speed: How fast the wait feels to a person. A growing answer feels faster than a blank box, even if the total time is the same.
Streaming: Sending and showing the answer in pieces as it is produced.
Buffering: Collecting data and sending it later in one lump. A buffering proxy turns a stream back into a single block.
Partial result: Whatever has arrived so far. Useful for text, useless for JSON.
Proxy and CDN: Servers that sit between the browser and your app and pass traffic along. Some collect a whole response before forwarding it, which is buffering.
Token: The small piece of text a model writes at a time: a short word, part of a longer word, or a punctuation mark. Topic 0.1 explains tokens.
Stand-in model: In this course, a scripted fake that answers with real timing so examples run without a key. A real app calls the provider from the server instead.

Step by step

01Two requests, two clocks

The first live example asks the same question twice against a stand-in model that takes about the same time either way. The first request is not streamed: nothing can be shown until the whole JSON object has arrived. The second is streamed: we note when the first piece of text arrives.

In a real app the route below lives on your server and calls the provider. Here serve(...) plays the server inside the page. Read the three lines at the end: the total time is about the same, but the first text arrives in a fraction of it.

Two requests, two clocksdiagram

Rendering diagram…

02What the user sees

Without streaming: a click, then a blank or spinning box for the whole total time, then all the text at once. With streaming: a click, a short wait, then text that grows while the user reads it. The model, the tokens and the bill are the same.

The practical rule: for any answer a person will read, measure and watch *time to first token*. It is the number that decides whether the feature feels alive.

03Streaming is not free

A streamed answer needs a reader loop (Topic 2.2), careful byte-to-text conversion (2.3), a parser for the wire format (2.4 and 2.5), rendering that keeps up (2.6), scrolling rules (2.7), and handling for a stream that stops halfway (Phase 3). Each is small. Together they are why teams reach for an SDK, and why it helps to have written the plain version once.

04When a stream is the wrong tool

The second example reads a JSON reply character by character and tries to parse the text after every piece. Of 52 pieces, only the last one gives valid JSON. A half-arrived object cannot be shown or used, so streaming it gains nothing and the extra code buys nothing.

It also shows two small edge cases: a one-word answer arrives as a single piece, and an empty answer arrives as no pieces at all. Your code must be ready for zero pieces.

When a stream is the wrong tooldiagram

Rendering diagram…

05Check the deployed site

A stream that works on your laptop can arrive as one block in production if something in front of your server buffers it. The headers below ask well-behaved proxies not to. Whether a platform honours them is documented by that platform, so test with a long answer on the real URL and watch whether text grows or appears at once.

app/api/chat/route.ts (real app)whole filets

// pass the provider's stream through to the browser without collecting it
return new Response(providerResponse.body, {
  headers: {
    "content-type": "text/event-stream; charset=utf-8",
    "cache-control": "no-cache, no-transform", // do not store or rewrite the stream
    "x-accel-buffering": "no", // nginx: do not buffer this response
  },
});

Try it yourself

1
Make the model slower
In the first example, change delayMs: 30 to 120 and run again. The total grows to about two seconds.
The two comparisons that matter stay true: the first text still arrives early, because it only has to wait for the first token. That is why a long answer from a slow model is still pleasant to use when it streams.
2
Count the pieces
In the second example, delete tokens: "chars", from the /api/order route. The stand-in now cuts the text at spaces, and this JSON has no spaces, so it arrives as one piece: readable after 1 pieces: 52 of 52 characters, then pieces: 1 | prefixes that parse: 1.
A chunk's size never changes what is possible: with 52 pieces or with 1, only the complete text is valid JSON.
3
Ask for a route that does not exist
In the second example's last loop, add "/api/missing" to the list. Nothing serves that path.
The fetch rejects with Uncaught TypeError: Failed to fetch, the same error as being offline. Whether you stream or not, a request can fail before the first byte, so "waiting" is not guaranteed to lead to "streaming". Phase 1 (Topic 1.3) showed what to say to the user for each failure.

Code & diagrams

Two clocks: first text and total time

Real timings vary from run to run, so the example logs comparisons instead of milliseconds. Change delayMs to 10 and run again: both totals shrink, and the first-text comparison stays true.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

not streamed: text shown once, after the whole answer: 79 characters
streamed: first text arrived before 1/3 of the total: true
streamed: total time about the same as not streamed: true
the user waits less before reading: true

When streaming does not help: JSON, one word, nothing

Only the complete text parses. A one-word answer is one piece, and an empty answer is zero pieces: never assume the reader loop runs at least once.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

readable after 52 pieces: 52 of 52 characters
pieces: 52 | prefixes that parse: 1
SyntaxError: Expected double-quoted property name in JSON at position 20 (line 1 column 21)
/api/yes -> 1 piece(s) "Yes."
/api/empty -> 0 piece(s) ""

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Parse JSON while it is still arriving

In the second example, delete the try { and } catch {} around JSON.parse(text) so the page parses every time a piece arrives and reads order.status from the result.

terminal

$ The console, on the very first piece

── what you'll see ──

Uncaught SyntaxError: Expected property name or '}' in JSON at position 1 (line 1 column 2)

Myth vs fact

Myth

Streaming makes the model faster.

Fact

The model writes at the same speed. Streaming only lets you show the text sooner, so the first words appear much earlier. The total time is nearly the same.

Myth

Always stream: it is always better.

Fact

Streaming adds states, parsing and failure cases. For a one-word answer, a classification, or JSON your code must read whole, a plain request is simpler and just as quick.

Myth

If it streams on my laptop it streams in production.

Fact

A proxy, CDN or host that buffers responses can turn your stream back into one block. Test the deployed URL with a long answer.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Log time to first token in production and watch its 95th percentile, not just its average. A feature with a good average and a slow tail still feels broken to the unlucky users. Typical causes of a slow first token: a very long prompt, queueing at the provider, a cold start of your own server function, and models that think before they answer.
▸
Stream for people, not for machines. A good rule when deciding per feature: if a human reads the text as it arrives, stream it. If code consumes the result, ask for the whole thing, or for structured output.
▸
A streamed answer starts to show before the model has finished, so you cannot withdraw it cleanly if a later check fails. If your product must check the whole answer first (for example for policy), either do not stream that feature or accept that some text will have been shown. Topic 9.6 returns to this trade-off.

Remember this

1
The problem this solves: A model writes its answer one token at a time and may need several seconds for a long one. If your page waits for the whole answer before showing anything, the user sees a blank box and wonders whether the click worked. This topic explains the two clocks that matter, what streaming changes and what it does not, and the cases where you should not stream at all.
2
Two clocks. *Time to first token* (TTFT) is how long the user waits before anything appears. *Total time* is how long until the answer is complete. Streaming leaves the total almost unchanged, because the model writes at the same speed either way. It shrinks the first clock from "the whole answer" to "the first few words". That is the only thing it does, and it is a great deal.
3
Why that feels faster. People read at roughly 200 to 250 words a minute, which is about four words a second. A model that writes faster than that stays ahead of the reader, so after the first words appear the user is never waiting again: they are reading while the rest is written. The wait did not disappear; it was hidden behind reading.
4
What streaming costs you. The answer is no longer one value that either arrives or fails. It is a sequence of pieces, so you must handle states in between (nothing yet, some text, finished), a connection that breaks halfway, and text the user has already read that you may later wish to retract. You also pay for the same tokens either way: streaming changes when you see them, not how many there are.
5
When not to stream. (1) *Very short answers*: "Yes." is one piece, so there is nothing to show early. (2) *Output you cannot use half-finished*: JSON is the main example, because {"order":"ORD-1042","sta cannot be parsed and cannot be shown. (3) *Work nobody watches*: a nightly summary saved to a database has no reader waiting. (4) *Answers you must check before anyone sees them*, for example a filter that has to read the whole text first. In these cases one ordinary request and one res.json() is simpler and just as good.
6
The same route can do both. Providers take a flag, usually stream: true in the request body. With it, the reply is a stream of small messages; without it, one JSON object. Your server decides which to ask for, and your page decides how to read the result. Topic 2.5 shows both shapes of the reply.
7
In production a stream can be silently turned back into a block. A proxy (a server that passes traffic along between the browser and your app), a CDN (a network of servers that sit in front of a site to deliver it faster) or a hosting platform that *buffers* responses collects your stream and delivers it all at once, and the user sees the blank box again. The usual fixes are response headers such as Cache-Control: no-cache, no-transform and, for nginx, X-Accel-Buffering: no, and checking your host's documentation for streaming support. Always test the deployed site, not just your laptop.

Explain it without notes

What does streaming change about an AI request, and what does it not change?

Why does a streamed answer feel faster even though the total time is about the same?

Name three cases where you should not stream, and what to do instead.

A streamed feature works locally but the user sees everything at once on the deployed site. What do you check?

Practice

Write a function shouldStream({ forHuman, isJson, expectedWords }) that returns true only when streaming is worth it, using the rules in this topic. Test it with four inputs.

Change the first example so it also logs the number of pieces the streamed answer arrived in. Is it the same as the number of words?

Trade-offs

↔
Streaming gives a fast-feeling start and costs code, states and partial-failure handling. For long prose answers the trade is clearly worth it; for short or structured results it is not.
↔
Passing the provider's stream straight through is the simplest server; reading and re-emitting it lets you filter, count tokens or stop early, at the price of more server code and the risk of buffering it yourself by accident.

Done when you can

I can explain time to first token and total time, and which one streaming improves.
I can say why a growing answer feels faster than a blank box.
I can name cases where I should not stream and what to do instead.
I know a zero-piece and a one-piece stream are possible, and my loop must cope.
I can name the usual reasons a stream arrives as one block in production.

Command Palette

Words you'll meet

Step by step

01Two requests, two clocks

02What the user sees

03Streaming is not free

04When a stream is the wrong tool

05Check the deployed site

Try it yourself

Make the model slower

Count the pieces

Ask for a route that does not exist

Code & diagrams

Break it on purpose

Parse JSON while it is still arriving

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can