PHASE 16Advanced ~26 min· topic 5 of 14

Topic 16.5

Streaming Responses with Generators and Async

In one line

Streaming sends a model's answer to you in small pieces as it is generated, so the user sees text appear instead of waiting for the whole reply. In Python this is a generator (or an async generator) that yields chunks, which you print, forward, collect and can cancel.

Think of it like this

A pizza shop that hands you each slice as it comes out of the oven, instead of making you wait until the whole pizza is done. You start eating sooner, and if you decide you are full after three slices, you can say "stop" and the kitchen stops baking. Streaming is slice-by-slice delivery of a model's answer.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Streaming: Receiving a result in small pieces as it is produced, instead of all at once at the end.
Chunk: One piece of a stream, usually a few characters or a word.
Time to first token: How long you wait before the first piece of the answer arrives.
Server-sent events (SSE): A way for a web server to keep one response open and send events into it over time.
Generator: A function with yield that produces values one at a time.
Async generator: An async def function with yield. You loop over it with async for.
Cancellation: Stopping a running task or stream early.

Step by step

01A stream is a generator

The simplest fake model is a generator that yields the scripted reply a word at a time. We fake it because a real model needs a key and a network, and a fake gives the same words every run. Recall from Topic 9.2: a generator gives out one value at a time and pauses between them. A real SDK stream is used the same way: a for loop that receives chunks. The first example prints each chunk with end="" and flush=True, so the text appears piece by piece, and also collects them.

main.pywhole filepython

def fake_stream(text):
    for word in text.split(" "):
        yield word + " "

for chunk in fake_stream("Streaming is slice by slice"):
    print(chunk, end="", flush=True)

02Collecting the final text

Streaming does not mean you lose the whole answer. Append each chunk to a list and join at the end. Joining a list is faster than adding strings repeatedly, and it keeps the chunk count for logging.

03Stopping early and cleaning up

When you break out of a for loop over a generator, the generator is paused, not finished. Calling .close() raises a special exit inside it at the yield, which runs its finally block. In the example, the finally block prints "[stream closed]". With a real stream that block would close the network connection.

Stopping early and cleaning updiagram

Rendering diagram…

04The async version

The second example is an async generator whose chunks arrive 0.3 seconds apart (a stand-in for network waits). The consumer uses async for inside async with asyncio.timeout(0.75). After the timeout the loop is cancelled, the generator's finally runs, and the code prints how many chunks arrived. asyncio.timeout is Python 3.11 and later.

real_stream_anthropic.pywhole filepython

with client.messages.stream(
    model="your-model-name",
    max_tokens=300,
    messages=[{"role": "user", "content": "Say hello."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

05Streaming to a browser

On a server, wrap your generator in a streaming response and let the framework write each chunk to the open HTTP connection. Browsers read it with the SSE or fetch-streaming APIs. Your Python function does not change: it still just yields strings.

A real streaming server also needs timeouts, a way to detect that the client left, and a way to report an error after some text has already been sent. That is why production chat systems add a final "done" or "error" event.

06Mistakes to avoid

Do not use a plain for on an async generator (the break-it example shows the error). Do not block the event loop inside a streaming handler with slow synchronous work. And do not forget that streamed text cannot be validated as JSON until the stream is complete (Topic 16.6): collect first, then parse.

Try it yourself

1
Cancel at a different place
In the first example change number == 3 to number == 6. Predict the printed text, then run.
2
Race the timeout
In the async example set the timeout to 0.05. How many chunks arrive? Where does the "[stream closed]" line appear relative to "timed out"?

Code & diagrams

A generator stream, collected and cancelled New tab

Notice that "[stream closed]" prints in both cases: the finally block runs when the stream ends and when you close it early.

Expected output

Streaming shows the answer while it is still being written. 
[stream closed]
final text: Streaming shows the answer while it is still being written.
chunks: 10
--- cancel after 3 chunks ---
Streaming shows the 
[stream closed]

An async stream with a timeout New tab

Change the timeout to 1.35 and run again to see more chunks arrive.

Expected output

got: one
got: two
[stream closed]
timed out after 2 chunks: one two

Streaming from FastAPI (not runnable here)python

A server-sent event is lines of text ended by a blank line, so a raw newline inside a chunk would end the event early. Encoding each chunk as JSON avoids that, and the browser decodes it with `JSON.parse`.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json

app = FastAPI()

@app.get("/chat")
async def chat(q: str):
    async def events():
        async for chunk in model_stream(q):      # your async generator
            # json.dumps keeps newlines inside a chunk from breaking the event format
            yield f"data: {json.dumps(chunk)}\n\n"   # one server-sent event
    return StreamingResponse(events(), media_type="text/event-stream")

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Loop over an async generator with a plain for

Use for instead of async for on an async generator.

main.pywhole filepython

import asyncio


async def tokens():
    yield "a"
    yield "b"


for t in tokens():
    print(t)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 9, in <module>

for t in tokens():

~~~~~~^^

TypeError: 'async_generator' object is not iterable

Myth vs fact

Myth

Streaming makes the model faster.

Fact

The model takes the same total time. Streaming only lets you show the early part sooner.

Myth

Breaking out of the loop stops the work on the server.

Fact

Only if the stream is really closed. Leaving a stream open can keep the connection, and your bill, running. Close it with a context manager or a finally block.

Myth

You can parse JSON from each chunk.

Fact

A chunk is a fragment. Join the whole stream, then parse. (Some libraries do incremental parsing, but that is advanced.)

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Async generators have their own cleanup rule: an abandoned one is closed by the event loop's asyncgen hooks at some later point, not immediately. Prefer contextlib.aclosing or an explicit await agen.aclose() when you stop early, as the practice shows.
▸
Back-pressure: if the consumer is slower than the producer, buffers grow. With a stream you pull one chunk at a time, so the producer waits. If you push chunks into an asyncio.Queue (Topic 12.8), give it a maximum size.
▸
Count tokens and measure time to first token in the streaming loop itself, and log them. They are the two numbers users feel. Do not use wall-clock time in tests: inject a fake clock.
▸
When a browser tab closes mid-answer, the web server (ASGI servers such as uvicorn, with Starlette) detects the disconnect and cancels the streaming task. Your generator then sees asyncio.CancelledError at its current await. Never swallow it: let it propagate through try/finally (or use async with on the upstream stream) so the upstream model request is closed. If you catch Exception broadly, remember that CancelledError is a BaseException since Python 3.8, so a plain except Exception does not catch it, but a bare except: does.
▸
A stream can fail after the HTTP status line (200) has already been sent, so you cannot turn it into a 500. The protocol you design must carry an in-band error or done event, and the client must treat a connection that ends without a done event as incomplete.

Remember this

1
Models produce text one token at a time. A full answer can take many seconds. Without streaming, the user stares at a blank screen until the end. With streaming, the first words appear quickly. The total time is the same; the time to first token and the feeling of speed are much better. Every chat product streams for this reason.
2
Over the web, streaming usually uses server-sent events (SSE): one HTTP response that stays open, and the server writes small text events into it as they are ready. The client reads them as they arrive. Provider SDKs hide the details and give you an iterator of chunks (an object a for loop can take values from one at a time; Topic 9.1); passing stream=True in an OpenAI-style call, or using client.messages.stream(...) with stream.text_stream in the Anthropic SDK, returns one.
3
A generator (Topic 9.2) is the natural Python shape for a stream. A function with yield hands out one chunk, pauses, and resumes when the caller asks for the next. A for loop over it prints chunks as they arrive, "".join(chunks) rebuilds the final text, and nothing is held in memory beyond the current chunk.
4
Real network streams wait for data, so they are async: an async generator (async def with yield, Topic 12.8) is consumed with async for. Recall from Topic 12.6: the event loop runs many async tasks by switching to another whenever one is waiting. While one stream waits for the next token, the event loop can serve other users, which is how one Python process streams to many people at once.
5
Cancelling matters. If the user closes the tab, or you hit a timeout, you must stop the stream so you stop paying for tokens nobody reads. For a normal generator, break out of the loop and call .close(); for an async one, asyncio.timeout(...) cancels the wait, and a try/finally inside the generator runs your cleanup (closing the connection) either way.
6
In a web service you do not collect the whole text: you pass each chunk on (for example into a FastAPI streaming response; Topic 15.5) and also keep a copy to save the final answer and count tokens. Errors can happen midway: the stream can break after half an answer, so decide in advance what to show (keep the partial text, mark it as incomplete) and whether to retry.

Explain it without notes

Why do chat products stream, and does it make the answer arrive sooner overall?

How does a generator model a stream, and how do you stop it early and clean up?

Why are real streams async, and how do you cancel one with a timeout?

The user closes the browser tab halfway through a streamed answer. What should happen in your Python service, and what goes wrong if it does not?

Practice

Write collect(stream) that returns the joined text and the number of chunks.

Write stream_until(stream, stop_word) that joins chunks until one contains the stop word, then closes the stream.

Write an async function first_n(agen, n) that returns the first n items of an async generator and closes it.

Trade-offs

↔
Stream or wait. Streaming improves perceived speed, but adds complexity: partial failures, cancellation, and no chance to validate the whole answer before showing it. For machine-to-machine calls that need a complete JSON object, do not stream.
↔
Sync generator or async. A plain generator is simpler and fine for scripts and tests. Servers handling many users should use async so one slow stream does not block the rest.

Done when you can

Done when you can explain why streaming improves perceived speed.
Done when you can write a generator stream and collect the final text.
Done when you can stop a stream early and show the cleanup running.
Done when you can consume an async generator with async for under asyncio.timeout.
Done when you know a full Hectal course on LLM engineering is planned to cover streaming UIs and production servers.

Command Palette

Words you'll meet

Step by step

01A stream is a generator

02Collecting the final text

03Stopping early and cleaning up

04The async version

05Streaming to a browser

06Mistakes to avoid

Try it yourself

Cancel at a different place

Race the timeout

Code & diagrams

Break it on purpose

Loop over an async generator with a plain for

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can