Topic 5.7
When a Stream Breaks Midway
In one line
A stream can end early with no error: the connection resets, a proxy times out, or the provider fails after sending some text. A reply is complete only when a finish reason arrived. Otherwise retry, show the partial text with a warning, and tell the user.
Think of it like this
A friend reading a letter aloud over the phone, and the line goes dead in the middle of a sentence. You do not pretend you heard the whole letter. You note what you got, call back, and maybe ask them to start again from the beginning.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Broken stream
- A stream that ended before the reply was complete, with or without an error.
- Finish reason
- The field that tells you the model finished. Without it a stream is not known to be complete.
- Idempotent
- Safe to do again: repeating it has the same effect as doing it once.
- Idempotency key
- A unique id sent with an action so a repeat of the same action is recognised and ignored.
- Connection reset
- The other side or something in between closed the connection abruptly.
- Read timeout
- A limit on how long to wait for the next data before treating the connection as dead.
Step by step
01Step 1: list how a stream can end
Finished (finish reason and [DONE]), dropped with an exception (connection reset, timeout), ended silently (no finish reason), or stopped by the user (Topic 5.5). Only the first is complete.
02Step 2: classify in one function
consume() returns the text read and a status. It catches the connection error and returns broken; after the loop it checks that a finish reason was seen. Nothing else in the program needs to know the details.
03Step 3: retry with a limit
ask() calls consume() up to two times. If an attempt is complete, it returns that text and says how many attempts it took. Otherwise it remembers the longest partial text.
04Step 4: fall back to a labelled partial reply
After the last attempt fails, the example reports PARTIAL with the status and the best partial text. A real UI shows it with a banner and a Retry button, not as a normal answer.
05Step 5: check for side effects
Before retrying a request that could call tools, check what already ran. Use idempotency keys for anything that changes the outside world. A broken stream during tool arguments has not run the tool, but a broken stream after a tool ran has.
06Step 6: measure
Count each ending by type. A rise in "ended silently" often means a proxy timeout, a rise in "connection reset" a network or provider incident. The cause tells you where to look.
Try it yourself
- 1
Allow a third attempt
Change
max_attemptsto 3 and give request 3 a third script that completes. What does the output say now? What would a real third attempt cost in tokens and seconds? - 2
Make a silent stream look finished
Change request 4 so its second attempt is complete. Then change
consumeto ignore the finish reason (remove the check). What does request 4 report now, and why is that the dangerous bug? - 3
Choose a policy
For each of three features (a chat answer, a tool call that sends an email, a translation of a paragraph), write which policy you would use: retry, show partial, or continue. Say why.
Code & diagrams
Four scripted requests: a clean one, a drop that a retry fixes, two drops, and a stream that ends silently. Java uses an Iterator that throws, since its examples stay single-threaded.
Expected output
request 1: complete after 1 attempt(s): Rain today
request 2: complete after 2 attempt(s): Rain today
request 3: PARTIAL (broken: connection reset), shown with a warning: Rain tod
request 4: PARTIAL (broken: ended without a finish reason), shown with a warning: Rain todBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust the end of the loop
In consume, delete the check that returns "broken: ended without a finish reason" (the if finish is None block; in JavaScript the if (finish === null) line; in Java the if (finish == null) line).
Myth vs fact
Myth
If no exception was raised, the reply is complete.
Fact
A silent end (a proxy closing the connection, a timeout) raises nothing. Check for a finish reason.
Myth
Retrying a stream is always safe.
Fact
It is safe for plain text. If a tool already ran, a repeat can do the action twice, so use idempotency keys and check what happened.
Myth
Partial text is useless after a failure.
Fact
It is real output. Show it with a warning, keep it in the history, or use it as the start of a continuation if your provider supports that.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Streams hide failures behind a 200 status, so monitoring by status code alone misses them. Log the ending of every stream with its type, tokens received and time, and alert on the share that are not
complete. - ▸
A retry after a late failure wastes the tokens already paid for. If most failures come late (after most of the reply), prefer the labelled partial reply plus a manual Retry for long answers, and automatic retry for short ones.
- ▸
Some failures are provider overloads that return an error event inside a 200 stream. Parse unknown events defensively: an object with an
errorkey is a break, not a chunk to assemble. Anthropic sendsevent: error(for example an overloaded error) inside an already-open stream, so a parser that keeps theevent:name can tell it apart from data. - ▸
Add a read timeout between events (for example 30 seconds). Combined with keep-alive comments from your relay (Topic 5.4), it separates "the model is thinking" from "the connection is dead".
Remember this
- 1
Why this exists. With a normal request, the failure is loud: an error status or an exception. A stream is different. The server already sent status 200 and part of the reply when something goes wrong, so the failure arrives as an exception in the middle of reading, or worse as a stream that simply stops. If your code treats "the loop ended" as "the reply is finished", you will show half a sentence as a final answer.
- 2
Know what complete means. Recall from Topic 5.3:
finish_reasonis the field that says why the model stopped. A reply is complete when you saw afinish_reason(and usually[DONE]as well). A stream that ends without a finish reason is broken, even if no exception was raised. The example classifies four requests: one clean, two that drop the connection (an exception), and one that just stops. The "just stops" case is the dangerous one. - 3
Choose a policy per feature. (1) Retry from the start: simplest and safest for text that nobody has acted on yet; discard the partial text, or keep it on screen until the new reply replaces it. You pay for the tokens of the failed attempt as well. (2) Show the partial text with a clear warning ("The answer was cut off") and a Retry button, when retrying automatically would surprise the user or cost too much. (3) Continue: send the partial text back as the start of the assistant's reply and ask the model to continue. Support for that differs per provider and the continuation can disagree with the earlier text, so treat it as advanced and test it.
- 4
Idempotency: is it safe to repeat? Repeating a text-only call is harmless. Repeating a call that already ran a tool (sent an email, created an order) is not. Before you retry a stream, check whether any side effect already happened, and give side-effecting actions an idempotency key so a repeat does nothing twice (Topic 12.4). Tool-call arguments are only complete after the stream finishes, so a stream that breaks mid-arguments has not run anything yet.
- 5
Tell the user. Silence is the worst option. Show that the reply was interrupted, keep what was received, and offer retry. Also tell your own monitoring: count broken streams by cause (connection reset, timeout, no finish reason, provider error event) and alert when the rate rises (Topic 11.4). Some providers send an error event inside the stream before closing; treat it as a break and show its message.
- 6
Limit retries and set a deadline. Use the same rules as ordinary calls: a small number of attempts (the example uses two), a pause with jitter between them (Topic 6.8), and an overall deadline so the user is not left waiting for five failed streams in a row. Also set read timeouts so a stream that goes silent is treated as dropped (Topic 2.7).
Explain it without notes
How do you decide whether a streamed reply is complete?
Describe three policies for a stream that breaks midway and when each fits.
Why does idempotency matter when retrying a stream?
Practice
Write classify_ending(finish_reason, raised) that returns "broken: error" if raised is true, "broken: no finish reason" if finish_reason is None, and "complete" otherwise.
Write best_partial(texts) that returns the longest string in a list (the first one if tied), or an empty string for an empty list.
Trade-offs
- ↔
Automatic retry hides failures from users but costs tokens and time; a visible partial reply is honest and cheap but needs a click.
- ↔
Continuing from a partial reply saves tokens and risks an inconsistent answer; restarting is simple and consistent.
- ↔
A short read timeout detects dead connections fast and can cut off slow reasoning models; tune it per model.
Done when you can
Done when you can classify a stream ending as complete or broken from its events.
Done when you can implement a retry limit that keeps the longest partial text.
Done when you can explain why side effects need idempotency keys.
Done when you can name what the user should see after a broken stream.