PHASE 5Intermediate ~22 min· topic 3 of 8

Topic 5.3

Assembling a Streamed Reply

In one line

A streamed reply arrives as many small deltas. An assembler adds the content deltas into text, joins tool-call argument fragments by their index, remembers the finish reason from near the end, and reads the token usage from the last chunk.

Think of it like this

A jigsaw puzzle delivered one piece at a time by post. Some pieces are sky, one has the box lid's picture, the last envelope holds a note saying "that was the final piece" and how many pieces there were. You cannot judge the picture from one piece, and you must not throw pieces away. You lay each one down in its place and only trust the picture when the note arrives.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Delta: The new part of the reply carried by one chunk (a few characters, or an argument fragment), not the whole reply so far.
Assembler: Code that turns a sequence of deltas back into one finished result.
finish_reason: Why the model stopped: stop (done), length (cut off at max_tokens), tool_calls (wants tools run), or others a provider defines.
Usage: The token counts for the request: prompt tokens, completion tokens and total.
Tool call: The model asking your program to run a named function with arguments (Phase 7).
Index: The number that says which tool call a fragment belongs to when several are streamed.
Fragment: A piece of the arguments text of a tool call. It is usually not valid JSON until joined with the others.

Step by step

01Step 1: read one chunk closely

A chunk is a JSON object with a choices list. Each choice has an index, a delta (what is new) and a finish_reason. In the first chunk the delta has the role. In most chunks it has a piece of content. In the stop chunk the delta is empty and the finish reason is set.

chunk.jsonwhole filejson

{"choices":[{"index":0,"delta":{"content":"Hel"},"finish_reason":null}]}

02Step 2: assemble text

Start with an empty string. For every choice in every chunk, append delta.content when it exists. After each chunk the example prints "text so far", which is exactly what a chat UI would show at that moment.

03Step 3: remember role, finish and usage

Variables for role, finish and usage start as None. A chunk sets them when it has them. The example's last chunk has "choices": [], so the loop over choices runs zero times and only the usage line matters.

Step 3: remember role, finish and usagediagram

Rendering diagram…

04Step 4: handle tool calls

Make a dictionary from index to a slot with id, name and arguments. For each tool-call fragment, find or create the slot for its index, set id and name when present, and append the arguments text. The second example streams two calls whose fragments are interleaved and still ends with two correct, separate calls.

05Step 5: parse only when complete

After the finish reason arrives (tool_calls), parse each slot's arguments with the JSON parser and validate them against the tool's schema (Topic 7.4). Running a tool with half of its arguments is a bug and can be a security problem.

main.pywhole filepython

import json

fragments = ['{"ci', 'ty":"Pune"}']
print(''.join(fragments))
print(json.loads(''.join(fragments))['city'])

06Step 6: package the result

Return one object: text, tool_calls, finish_reason, usage. The rest of the program, which was written for non-streaming calls, can then use the same shape. The playbook topic (5.8) builds this as a reusable class.

Try it yourself

1
Print every delta
In the first example, print delta.get("content") before adding it. Which chunks give an empty string, and which give nothing at all (None or null)? Why does the code use or ""?
2
Move the usage
Move the usage object into the stop chunk ("finish_reason":"stop" plus "usage":{...}) and delete the last chunk. The loop should still print the same result. This is the shape some providers use.
3
Reverse the fragments
In the second example, swap the order of the two fragments for call 0 ('ty":"Pune"}' before '{"ci'). What does the program do when it parses the joined text? Network order is not guaranteed to be wrong, but a parser should never trust fragments parsed alone.
4
A third call
Add a call with index 2 named get_time and arguments {"city":"Goa"} in three fragments. Does the output list it last? Which line of the code decides the order?

Code & diagrams

Assembling text, the stop reason and usage New tab

Six scripted chunks: role only, three text deltas, the stop chunk and a final chunk with the token counts and no choices.

Expected output

chunk 1: text so far = []
chunk 2: text so far = [Hel]
chunk 3: text so far = [Hello]
chunk 4: text so far = [Hello!]
chunk 5: text so far = [Hello!]
chunk 6: text so far = [Hello!]
role: assistant
finish_reason: stop
usage: prompt=12 completion=3 total=15

Joining tool-call fragments by index New tab

Two parallel calls, with their argument text cut into fragments that arrive interleaved. Java builds the JSON with small Map helpers.

Expected output

finish_reason: tool_calls
call 0 (call_a): get_weather(Pune)
   raw arguments: {"city":"Pune"}
call 1 (call_b): get_weather(Delhi)
   raw arguments: {"city":"Delhi"}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Replace instead of append

In the first example, change text += delta.get("content") or "" to text = delta.get("content") or "" (in JavaScript text = delta.content ?? ""; in Java text = ...).

terminal

$ Run

── what you'll see ──

chunk 1: text so far = []

chunk 2: text so far = [Hel]

chunk 3: text so far = [lo]

chunk 4: text so far = [!]

chunk 5: text so far = []

chunk 6: text so far = []

role: assistant

finish_reason: stop

usage: prompt=12 completion=3 total=15

Myth vs fact

Myth

Each chunk contains the reply so far.

Fact

Chat streams send deltas, only the new part. You build the full reply yourself by appending.

Myth

Every chunk has one choice with a content string.

Fact

The first chunk may carry only the role, the stop chunk has an empty delta, tool-call chunks have content: null, and a usage chunk can have no choices.

Myth

Each tool-call fragment is a small JSON object I can parse right away.

Fact

Fragments are cut at arbitrary points of the arguments text. Only the joined text is JSON, and only after the model has finished.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Streams of tool calls need a rule for the moment of truth: execute only after finish_reason is tool_calls and the joined arguments validate against the tool's schema. Some UIs show "Calling get_weather..." as soon as the name arrives, and run the tool later.
▸
finish_reason: "length" on a streamed reply means the text stops mid-sentence with no error. Surface it to the user and to your metrics. A tool-call stream that ends with length has truncated arguments that will not parse (Topic 4.6 covers repair).
▸
Usage numbers drive billing and budgets (Phase 6). If your provider only reports usage when asked, set that option, and when the usage chunk never arrives (a cancelled or broken stream), estimate tokens from the text and mark the figure as an estimate.
▸
Keep the assembler a pure class with feed(event) and result(): no network, no UI. It then runs unchanged against live streams, recorded streams in tests, and fakes in this course.

Remember this

1
Why this exists. The SSE parser from Topic 5.2 gives you events. A program needs a finished result: the whole text, the tool calls to run, why the model stopped and what it cost. The pieces are deltas: each one holds only what is new, not the reply so far. An assembler is the small piece of code that turns the sequence of deltas back into one result.
2
Text: add the content deltas. Each chunk has choices[0].delta. The first chunk often carries only "role": "assistant" and an empty content. Later chunks carry content pieces, which you append in order. Some deltas have no content key (the stop chunk) or content: null (tool-call chunks), so read it as delta.get("content") or "". The break-it below replaces instead of appends and shows how little survives.
3
The stop reason comes near the end. finish_reason is null in every chunk until the last content chunk, which has an empty delta and a value such as "stop" (finished normally), "length" (hit max_tokens, so the text is cut off: Topic 2.3 shows it in a normal response), or "tool_calls" (the model wants tools run). Remember the first non-null value. A stream that ends without any finish reason is broken (Topic 5.7).
4
Usage comes last, and where it lives depends on the provider. The token counts (prompt_tokens, completion_tokens, total_tokens) are only known once the model is done. OpenAI-style APIs send them in an extra final chunk with an empty choices list, and only when you ask for it (stream_options: {"include_usage": true}). Some providers also put usage under a provider-specific key: Groq (the lab's provider, checked on 2026-10-11) always adds it under x_groq.usage in the stop chunk, and sends the standard extra chunk as well when you set stream_options. The example handles the extra-chunk shape; turn on Stream in a lab and read the last raw lines to see it. Code that does chunk["choices"][0] on every chunk crashes on that last chunk, which is why the example loops over choices instead.
5
Tool calls: join fragments by index. When the model calls a function, the arguments are JSON text that arrives in fragments such as {"ci, then ty":"Pune"}. Each fragment sits in delta.tool_calls[] with an index. The first fragment for an index also carries the id and the function name. Later ones carry only more arguments text. You keep one slot per index and append: no fragment is valid JSON alone, so parse the text only after the finish reason arrives. Two parallel calls can interleave, so never assume the fragments come in order of call.
6
Keep the raw pieces for debugging. Store the list of raw events (in tests and in a debug log) next to the assembled result. When a user reports a broken answer you can replay the exact stream through the assembler. This is the same idea as the fake streams in this course: a stream recorded once becomes a repeatable test (Phase 9 builds on this).

Explain it without notes

Explain why a streamed reply uses deltas, and what an assembler does with them.

How do you assemble streamed tool calls, and why must you not parse a fragment on its own?

Where do finish_reason and usage appear in a stream, and what should you do if they are missing?

Why does chunk["choices"][0]["delta"] crash on some real streams?

Practice

Write join_text(chunks) that takes a list of chunk dictionaries (each like {"choices": [{"delta": {...}}]}) and returns the joined content text, skipping chunks with no choices or no content.

Write join_arguments(fragments) that takes a list of (index, text) pairs and returns a dictionary from index to the joined text, in the order the fragments arrived.

Trade-offs

↔
A pure assembler class is simple to test but one more object to carry; calling feed straight from the reader loop is shorter and harder to reuse.
↔
Showing a tool call as soon as its name arrives is responsive, but the arguments may still change before you can run it.
↔
Keeping raw events costs memory and may include user data; keep them in tests and short-lived debug logs, not forever.

Done when you can

Done when you can write an assembler that returns text, finish reason and usage from a list of chunks.
Done when you can explain why a chunk can have no choices, no content or null content.
Done when you can join tool-call fragments by index and parse them only when complete.
Done when you can say what a missing finish reason or missing usage means.

Command Palette

Words you'll meet

Step by step

01Step 1: read one chunk closely

02Step 2: assemble text

03Step 3: remember role, finish and usage

04Step 4: handle tool calls

05Step 5: parse only when complete

06Step 6: package the result

Try it yourself

Print every delta

Move the usage

Reverse the fragments

A third call

Code & diagrams

Break it on purpose

Replace instead of append

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can