Command Palette

Search for a command to run...

Hectal
Phase 5Intermediate6 of 15 in LLM Engineering

Streaming and UX

Show the first words in under a second: server-sent events, assembling a streamed reply, cancelling, partial JSON and streams that break midway.

A model writes one token at a time, and a long answer takes many seconds. A screen that stays blank until the last token arrives feels broken, even when the total time is the same. Streaming sends each piece as soon as it exists, so the reader starts reading after a fraction of a second. This phase teaches the machinery underneath: the server-sent events wire format, a parser built by hand, and the assembler that turns many small chunks back into text, tool calls, a stop reason and a token count.

Streaming also creates new ways to fail. A chunk can end in the middle of a line. A proxy can buffer everything and defeat the point. The user can press Stop and you keep paying for tokens nobody reads. The connection can drop with half a reply on screen. You will build each piece with a fake model (so every example is free, repeatable and runs in Python, Java and JavaScript) and use real-model labs to see the raw lines a real provider sends and the time to first token.

The phase ends with a playbook: an SSE parser, a stream assembler and a cancel-safe consumer that you can copy into a product. It assumes Phase 2 (the HTTP API under every SDK) and Phase 4 (JSON and validation).

0/8 · 0%
8 topics ~3 h 18 code blocks & diagrams
Start with the first topic
1
5.1

Why Stream

Streaming does not make the model faster. It shows the first words after the model starts writing instead of after it finishes, which is what the reader feels. It does nothing for programs that need the whole reply, such as JSON for code.

22 min 3 code practice

2
5.2

Server-Sent Events from First Principles

Streaming APIs send server-sent events: lines that start with data:, separated by blank lines, ending with [DONE]. Network chunks can cut a line anywhere, so a parser must buffer until the blank line.

24 min 4 code practice

3
5.3

Assembling a Streamed Reply

A streamed reply arrives as many small deltas. An assembler adds the content deltas into text, joins tool-call argument fragments by their index, remembers the finish reason from near the end, and reads the token usage from the last chunk.

22 min 2 code practice

4
5.4

Relaying a Stream to the Browser

Your server sits between the provider and the browser, so it must pass the stream along as it arrives. Buffering proxies, compression and wrong headers can silently turn a stream into one late lump.

21 min 2 code practice

5
5.5

Cancelling a Stream

When the user presses Stop or leaves the page, close the stream so the provider can stop writing and your program frees its resources. Hiding the text while you keep reading does not stop anything.

20 min 2 code practice

6
5.6

Streaming Structured Output

Half of a JSON document is not valid JSON, so a normal parser fails until the end. A partial-JSON reader closes the open string and brackets so you can show fields as they arrive, for display only, never for decisions.

21 min 1 code practice

7
5.7

When a Stream Breaks Midway

A stream can end early with no error: the connection resets, a proxy times out, or the provider fails after sending some text. A reply is complete only when a finish reason arrived. Otherwise retry, show the partial text with a warning, and tell the user.

20 min 1 code practice

8
5.8

Playbook: Streaming Recipes

Three recipes to copy: an SSE parser, a stream assembler, and a cancel-safe consumer that always ends in a known state and always cleans up. Plus the checklist to run before you ship a streaming feature.

22 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Write the SSE parser

Write sse_events(chunks). chunks is a list of text pieces cut anywhere from an SSE stream. Return the list of data strings of every complete event, in order.

Intermediate
2

Assemble a streamed reply

Write assemble(events). events is a list of data strings: JSON chunks, and possibly "[DONE]". Return a dictionary {"text": ..., "finish_reason": ..., "usage": ...}.

Intermediate
3

Join streamed tool calls

Write join_tool_calls(events). Each event is a JSON chunk string. Tool-call fragments are in choices[0].delta.tool_calls, each with an index. The first fragment of an index also has function.name; every fragment may have function.arguments (a piece of JSON text).

Intermediate
4

Stop reading and close the stream

Write read_until_stop(stream, should_stop). stream is a generator of text pieces. Read pieces and join them into text. After adding each piece, call should_stop(text); when it returns True, stop reading.

Intermediate
5

Is the stream complete?

Write stream_status(events). events is an iterable of data strings (JSON chunks, maybe "[DONE]"). Iterating it may raise ConnectionError partway.

Intermediate