PHASE 5Intermediate ~22 min· topic 8 of 8

Topic 5.8

Playbook: Streaming Recipes

In one line

Three recipes to copy: an SSE parser, a stream assembler, and a cancel-safe consumer that always ends in a known state and always cleans up. Plus the checklist to run before you ship a streaming feature.

Think of it like this

A pilot's pre-flight checklist. Each item is small and obvious, and the list exists because the one item you skip is the one that bites. The recipes below are the items; the checklist at the end is the list.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Recipe: A small, tested piece of code you can copy into a product.
Replay: Feeding a recorded stream through your code again, to test or debug it.
Terminal state: A state in which the stream is over: finished, stopped or broken.
Pre-flight checklist: A list you go through before shipping, so common mistakes are caught.
Raw events: The exact event strings from the wire, before you assemble them.

Step by step

01Step 1: parser and assembler as two small classes

The first example defines SseParser (text chunks in, event strings out) and StreamAssembler (event strings in, finished reply out). They do not know about each other, or about the network.

02Step 2: join them in read_stream

read_stream feeds chunks to the parser and events to the assembler until the assembler says [DONE]. If the chunks run out first it raises StreamEnded with the partial text.

Step 2: join them in read_streamdiagram

Rendering diagram…

03Step 3: run three streams, cut every 7 characters

A text stream with usage, a tool-call stream and a stream that ends early are cut into 7-character chunks. The first two give complete results; the third raises.

04Step 4: the cancel-safe consumer

The second example consumes a fake stream with a simulated clock. It tracks the time of the first and last pieces, handles Stop at 520 ms and a drop at 550 ms, and counts cleanups: one per run, whichever way it ended.

05Step 5: plug in the real network

The shown code feeds response.iter_text() (Python), a text-decoded response.body (JavaScript) or BodyHandlers.ofLines() (Java) into the same recipes. Only this part touches the network.

06Step 6: run the checklist

Before shipping, go through the checklist below. Each item matches a failure from this phase, and each can be tested with a fake stream.

Try it yourself

1
Cut it everywhere
In the first example change the chunk size 7 to 1, 3 and 1000. The three result lines must stay identical. If one changes, your parser has a boundary bug.
2
Replay a broken stream
Add a fourth stream to the first example: a text stream with [DONE] but no finish_reason. What does read_stream return, and what should it return? Add a check in read_stream so it raises StreamEnded in this case too.
3
Stop earlier
In the second example change the stop time from 520 to 470. Predict the text and the last time, then run it.
4
Run it for real
Use the earlier labs with streaming on, and compare the time to first token the lab shows with the first= value of the consumer. They measure the same thing.

Code & diagrams

Recipes: SSE parser, stream assembler and read_stream New tab

Three recorded streams, each cut into 7-character chunks: text with usage, a tool call, and a stream that ends before [DONE].

Expected output

text stream: text=[Hello!] finish=stop total_tokens=15 calls=-
tool stream: text=[] finish=tool_calls total_tokens=- calls=get_time({})
cut stream : error: stream ended before [DONE] (text so far: Hel)

Recipe: a cancel-safe consumer with time to first token New tab

Time is simulated. The same consumer ends as finished, stopped or broken, and cleans up exactly once each time.

Expected output

finished: text=[Hello world] first=450 ms, last=600 ms, cleanups=1
stopped : text=[Hello] first=450 ms, last=520 ms, cleanups=1
broken  : text=[Hello] first=450 ms, last=500 ms, cleanups=1

Where the real stream plugs inpython

Not run here: they need a network and your own GROQ_API_KEY. Only these lines touch the network; the recipes above do the rest.

# The recipes from this topic (SseParser, StreamAssembler, read_stream) plug in after the network call.
# pip install httpx
import os

import httpx

body = {
    "model": "qwen/qwen3.8-27b",
    "stream": True,
    "messages": [{"role": "user", "content": "Say hello in five words."}],
}
headers = {"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]}  # the key comes from the environment

with httpx.stream("POST", "https://api.groq.com/openai/v1/chat/completions", headers=headers, json=body, timeout=30) as response:
    response.raise_for_status()
    try:
        reply = read_stream(response.iter_text())  # iter_text decodes UTF-8 across chunk boundaries for you
    except StreamEnded as error:
        reply = {"text": error.partial, "finish": "broken"}  # show the partial text with a warning (Topic 5.7)
print(reply["text"], reply["finish"])
# With the official openai package the same stream is an iterator of typed chunks (stream=True). Pass
# stream_options={"include_usage": True} to ask for the final usage chunk (OpenAI and Groq support it; check other providers).

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Skip the cleanup

In the consumer, remove the cleanup in the finally block: replace stream.close() with pass (JavaScript: delete stream.return();; Java: delete stream.close();).

terminal

$ Run

── what you'll see ──

finished: text=[Hello world] first=450 ms, last=600 ms, cleanups=1

stopped : text=[Hello] first=450 ms, last=520 ms, cleanups=0

broken : text=[Hello] first=450 ms, last=500 ms, cleanups=1

Myth vs fact

Myth

The recipes are only for teaching, real projects use an SDK.

Fact

SDKs hide the parser and assembler, but the checks (finish reason, cancel, cleanup, no action on partial data) are yours either way.

Myth

One test with a fast local stream is enough.

Fact

Test with chunks cut at every size, with drops, with silent ends and with Stop at different moments.

Myth

Streaming code should live next to the UI code.

Fact

Keep the parser and assembler pure so fake and recorded streams can test them without a browser or a network.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Make recordings: save the raw events of real streams (with user data removed) as fixtures. They are the best regression tests for the assembler, and the lab's raw view is a quick way to capture one.
▸
Put the recipes behind one interface, for example stream_chat(messages) returning events of type text, tool_call, done and error. Then UI code, logging and retries are the same for every provider.
▸
Instrument at the consumer: time to first token, tokens per second, ending type, and cleanup count. These four numbers answer most "why does chat feel slow" questions (Topic 11.4).
▸
Review each streaming route for the platform: maximum duration, concurrent connections, response buffering and compression. Many production incidents are a platform default, not a code bug.

Remember this

1
Recipe: SSE parser. When to use it: whenever you read a stream yourself instead of through an SDK. It guards against chunks that cut a line anywhere. Adapt it by accepting \r\n, by keeping event: and id: fields when your provider uses them, and by feeding it decoded text from a streaming decoder. Test it by cutting a known stream at every size.
2
Recipe: stream assembler. When to use it: to turn events into one result with text, calls, finish and usage. It guards against deltas treated as snapshots, empty choices, null content and tool-call fragments parsed alone. Adapt it for your provider's usage location, and keep the raw events in tests. Keep it free of network and UI code so a recorded stream can replay through it.
3
Recipe: read_stream. When to use it: the three-line loop that joins both. It guards against the silent ending: if the bytes end before [DONE], it raises StreamEnded carrying the partial text, so the caller must decide what to do. Never return a result from a stream that did not finish.
4
Recipe: cancel-safe consumer. When to use it: every place a person can press Stop or leave. It guards against leaks and against paying for text nobody reads. It always ends in one of three states (finished, stopped, broken), records the time to first token and the last token time, and closes the stream in finally. Adapt it by adding a deadline and the retry policy from Topic 5.7.
5
Recipe: relay with the right headers. When to use it: your server sits between the browser and the provider. It guards against buffering proxies and abandoned provider calls: send text/event-stream (or plain text pieces), Cache-Control: no-cache, X-Accel-Buffering: no for nginx, pass cancellation upstream, and test the deployed route with curl -N.
6
Recipe: preview but do not act. When to use it: structured output on screen. It guards against decisions on partial data. Show the live view for effect, the stable view for anything the user can click, and parse and validate the complete text before any action (Topic 5.6).

Explain it without notes

Walk through what happens to one chunk of bytes from the network until it becomes text on the screen.

Which three states can a cancel-safe consumer end in, and what must always happen?

Give five items you would put on a pre-flight checklist for a streaming feature.

Practice

Write chunked(text, size) that cuts a string into pieces of size characters (the last one may be shorter), and use it to check that a parser gives the same events for sizes 1, 3 and 1000. Use the SseParser below.

Write summarise(result) that turns a result dictionary with text, finish and usage (or None) into one line such as text=[Hi] finish=stop tokens=15, writing tokens=- when usage is None.

Trade-offs

↔
Writing the recipes yourself gives control and testability; an SDK gives speed and updates. Many teams use the SDK and keep their own consumer and checks around it.
↔
Strict completeness checks reject odd but harmless streams (Anthropic's stream has no [DONE] and ends with a message_stop event, so the [DONE] rule in these recipes must be changed for it); tune the rule to what your provider really sends.
↔
More instrumentation costs a little code and a little storage, and pays back at the first incident.

Done when you can

Before you ship: the SSE parser gives the same events at chunk sizes 1, 7 and 1000, tested with a fake stream.
Before you ship: a reply counts as complete only with a finish reason; every other ending is classified as broken and handled.
Before you ship: usage is requested and read, or estimated and marked as an estimate.
Before you ship: tool arguments are parsed and validated only after the stream ends; partial JSON is used for display only.
Before you ship: Stop closes the stream, cancellation reaches the provider, cleanup runs in finally, and open streams returns to zero in tests.
Before you ship: the deployed route streams through every proxy (tested with curl -N), with headers set, keep-alives on and platform limits checked.
Before you ship: time to first token, total time and the ending type of every stream are logged.

Command Palette

Words you'll meet

Step by step

01Step 1: parser and assembler as two small classes

02Step 2: join them in read_stream

03Step 3: run three streams, cut every 7 characters

04Step 4: the cancel-safe consumer

05Step 5: plug in the real network

06Step 6: run the checklist

Try it yourself

Cut it everywhere

Replay a broken stream

Stop earlier

Run it for real

Code & diagrams

Break it on purpose

Skip the cleanup

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can