PHASE 2Beginner ~38 min· topic 8 of 8

Topic 2.8

Build: Mitra Streams

In one line

Mitra now reads the real chat stream format with a parser that survives every split, shows the answer growing at most once per frame, scrolls politely, and tells apart a finished answer, a cut-short one, a broken one and an empty one.

Think of it like this

The market stall from Topic 0.5, a few weeks later. The owner already has the locked till and the "back in 5 minutes" signs (Mitra v1). Now orders come in over a proper phone line, which crackles and cuts words in half. A good assistant on that line writes down what it hears, waits for the end of each sentence, and says "sorry, the line dropped" when it does. Mitra v2 keeps everything v1 learned and adds that careful listening.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

readSSE: Mitra's helper: an async generator that turns a streamed response into text, finish and error parts.
askMitra: Mitra's helper that sends the request, limits the wait for the response headers, checks res.ok and throws a MitraError, so a failure status never reaches the reader.
MitraError: Mitra's own error type from Topic 1.6, with a kind such as timeout or rate_limited. It never carries the provider's words.
Response headers: The first part of an HTTP response (status and header lines). fetch resolves when they arrive; the body streams in afterwards.
role="log": An accessibility role for a region where new messages are added at the end, like a chat.
Stand-in route: In these pages, fakeOpenAI plays the server. In a real app it is your route handler calling the provider.
Finish note: A small line under an answer that explains why it ended unusually, such as length.
Scripted visitor: Lines at the end of an example that type and click for you.

Step by step

01The route, now in the real format

The route is v1's route with one change at the end. The stand-in provider (fakeOpenAI with a reply function) answers by keyword: cut breaks the stream after some text, long ends with finish_reason: length, flaky fails once with a 503, busy is a 429 with retry-after, slow makes the provider wait two seconds before it answers at all. It cuts the stream into 17-byte packets, so every event arrives in pieces.

Where v1 turned the provider's JSON into its own JSON, v2 hands the provider's stream to the browser untouched. A route that only forwards a stream is the simplest server there is (Topic 2.1's trade-off), and it is where cache-control: no-cache, no-transform goes.

the end of the routewhole filejs

if (!upstream.ok) { /* log the provider's reason, return only a code (as in v1) */ }
// pass the provider's stream straight through, and ask proxies not to buffer or rewrite it
return new Response(upstream.body, {
  headers: { "content-type": "text/event-stream; charset=utf-8", "cache-control": "no-cache, no-transform" },
});

02Reading and drawing

run ignores a second call while one is busy, goes to waiting, calls askMitra, and loops over readSSE. The first text part moves the state to streaming. Each text part is added to a local variable and a frame is scheduled if none is pending. A finish part with length or content_filter sets a note. After the loop it cancels the frame and dispatches done with the final text.

The wait for response headers is limited by an AbortController and a timer that is cleared as soon as fetch resolves. The stream that follows is not limited at all: a long answer must be allowed to take as long as it needs.

03Failing, and retrying only while nothing has been shown

One catch handles all shapes: a failure status or timeout from askMitra, a thrown error from a broken connection, and an error event that readSSE turned into a MitraError. If nothing has been shown and the kind is worth retrying (server, timeout, offline), it waits a moment and tries again, up to three attempts. Otherwise it stops. If some text arrived, the message says the answer was cut off and the text stays. Either way the status becomes error, which unlocks the form.

Why not retry after the first word? The second attempt would write a different answer, and the reader has already started the first. Topic 3.4 treats retry and regenerate properly.

Failing, and retrying only while nothing has been showndiagram

Rendering diagram…

04Every kind of answer, as a timeline

The second example does not draw anything. It sends seven kinds of question and prints the states each goes through: a normal answer, an empty one, one cut by an error, one cut by length, a busy server (429), one with an emoji and Hindi cut into 11-byte packets, and a request that cannot reach any server (offline). Use it as the test list for the parser.

05Watch it run

The first example is the whole of Mitra v2, with a scripted visitor that asks seven questions, each when the previous one has finished: shipping, a refund, a provider that fails once (attempt 1 failed (server), trying again, then an answer), a stream that breaks, a long policy that hits the length limit, a provider that is slow to start (two retries, then "taking too long"), and a busy one (429, no retry, a wait time). Read the console top to bottom, then look at the page and type your own.

Try it yourself

1
Teach the stand-in a new answer
Add [/hours|open/i, "We answer messages from 9 am to 6 pm, Monday to Saturday."] to ANSWERS and ask about opening hours. The screen code does not change: it does not care where the words come from.
2
Make the packets tiny
Change splitBytes: 17 to 1. The answers still arrive intact, including the ☕. Change it to 0 and nothing visible changes either, which is the point of buffering.
3
Scroll up mid-answer
Type 'Tell me the long policy' into the box and scroll up in the message box while it streams. The Jump to latest button appears and the box stays where you put it.
4
Never give up, then give up
Change attempt >= 3 to attempt >= 1 and run: the flaky question now fails at once with failed: Mitra had a problem on its side. Please try again., because no retry is allowed. Change it to attempt >= 6 and the slow question takes noticeably longer to fail (five timeouts). A retry limit trades patience against waiting time and load on a struggling provider (Topic 1.4).

Code & diagrams

Mitra v2

A scripted visitor asks seven questions. Then ask your own: try 'What tea do you sell?', an empty question, or the word crash. The stream is cut into 17-byte packets, and the ☕ in the refund answer still arrives whole. The (server log) lines are what only you, the developer, see.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

answered: "Orders ship in 1 to 2 working days. Order ORD-1042 (Masala chai 250 g) has shipped."
answered: "You can return any unopened item within 7 days of delivery. Returns are free ☕"
(server log) provider said 503 The engine is overloaded.
attempt 1 failed (server), trying again
answered: "Orders ship in 1 to 2 working days. Order ORD-1042 (Masala chai 250 g) has shipped."
failed: cut off after "Your order ORD-1043 is being packed and will"
answered: "Our full returns policy has many parts. Part one is about unopened items and covers"
attempt 1 failed (timeout), trying again
attempt 2 failed (timeout), trying again
failed: Mitra is taking too long. Please try again.
(server log) provider said 429 Rate limit reached.
failed: Mitra is busy. Try again in 2 seconds.

The states for every kind of answer

This is the list to test any new parser against. The 'cut' case works even though the stand-in breaks the connection after the text: the loop throws after part of the answer has arrived. The 'offline' case is a fetch that cannot reach any server: it rejects with a TypeError before any response exists.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

normal: waiting -> streaming -> done
empty: waiting -> done (empty answer: show a message)
cut: waiting -> streaming -> error (partial text kept: "Your order ORD-1043 is being")
long: waiting -> streaming -> done + note: cut short by the length limit
busy: waiting -> error (busy)
emoji: waiting -> streaming -> done
offline: waiting -> error (offline)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Skip the res.ok check

In askMitra, change if (!res.ok) { to if (false) {. Run and look at the last console line, for the busy question.

terminal

$ The last console line

── what you'll see ──

answered: ""

Myth vs fact

Myth

If the stream parsed without errors, the answer is complete.

Fact

A stream can end quietly with no finish reason, or a failure body can parse to nothing. Check the finish reason and the status.

Myth

Tests with English text and big chunks are enough.

Fact

The bugs of this phase appear with emoji, Hindi and small packets. Test with them.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep readSSE and askMitra in their own file. Everything provider-specific stays there, so Phase 5 can replace them with an SDK and the components do not change.
▸
Keep the list of answers in the second example as a permanent test table. Every new failure you meet in production becomes a new row.
▸
Mitra still sends only the last question. It has no memory yet; Phase 4 adds the conversation and decides how much history to send.

Remember this

1
The problem this solves: Mitra v1 (Topic 1.6) fails well, but it waits for a whole JSON answer before showing anything. Real providers stream the chunk format of Topic 2.5 over SSE, cut into packets, with emoji and Hindi inside. This topic puts Phase 2 on top of v1: the same typed errors, time limit and retry, but the answer now arrives through readSSE, is drawn once per frame, and is scrolled politely.
2
Starting point: Mitra v1. It has the route /api/chat (validates the body, adds the system message, hides provider errors), MitraError with the kinds bad_request, rate_limited, server, timeout and offline, askMitra(messages, { signal }), a reducer, a ref guard against double sends, and a retry for the failures worth retrying. The provider is the fakeOpenAI stand-in. In a real app the one block that changes is the provider call inside the route, which would use your key (Topic 1.5).
3
What changes from v1. (1) The route asks the provider for stream: true and passes the stream through with content-type: text/event-stream and cache-control: no-cache, no-transform (Topic 2.1). (2) askMitra now returns the Response instead of res.json(), and its time limit covers only the wait until the response headers arrive: a limit on the whole call would abort a long answer that is streaming normally. A stream that starts and then stalls needs a watchdog, which is Topic 3.5. (3) readSSE(response) from Topic 2.5 yields text, finish and error parts; an error event becomes a MitraError, so the provider's words stay in the server log. (4) The reducer gets streaming and text actions. (5) A retry is allowed only while nothing has been shown: after the first word, a retry would produce a different answer, so a failure then keeps the partial text and says it was cut off (Topic 3.3 builds on this).
4
The screen. The message box has a fixed height and scrolls inside itself, with role="log". New text follows only a reader at the bottom, and a Jump to latest button appears otherwise (Topic 2.7). The growing answer is stored in a variable and drawn at most once per frame, with the last piece drawn when the stream ends (Topic 2.6). The element ids are the registry's: #messages, #prompt, #send, #status and #error.
5
Every scenario this version handles. *Normal:* tokens grow with correct text. *Emoji and Hindi cut across packets:* intact, because of the decoder. *Packets cutting lines:* the buffer. *finish_reason: length:* the text stays and a note says it was cut short. *content_filter:* a note says Mitra cannot answer that. *An empty answer:* a message instead of a blank bubble. *An error event or a dropped connection after some text:* the partial text stays, marked as cut off, and no retry. *A 503 once, then fine:* retried, the user sees "Trying again". *A provider that is slow to start:* a timeout after the header limit, retried, then a message. *A 429:* a message with the wait time, not retried. *A 500 or no connection before any text:* a message a person can act on, and the form unlocks. *A double click:* ignored by the ref guard.
6
What Mitra can do now that it could not before: read the standard provider stream whatever the packet sizes, stay correct with non-English text and emoji, render long answers without stutter, let a reader scroll up while an answer is still arriving, and still fail well. What is still missing, and where it is added: a Stop button, partial-answer controls, retry and regenerate, a stall watchdog and a rate-limit countdown (Phase 3); a conversation with memory (Phase 4); an SDK in place of this plumbing (Phase 5); safe formatted output (Phase 6); cards and tools with approvals (Phases 7 and 8); and the rest in later phases.

Explain it without notes

Walk through what happens from the model writing a token to the pixel appearing in Mitra v2.

Which failure and edge cases does Mitra v2 handle, and how is each shown?

Why is readSSE a separate function from the component?

What is still missing from Mitra, and which phase adds it?

Practice

Add a visible Finished: length or Finished: stop line under each answer using the finish reason.

Make the empty-answer message appear only after the stream ended, not while waiting.

Trade-offs

↔
A hand-written reader gives full control and teaches the format; an SDK replaces it in Phase 5. Knowing this version makes debugging the SDK easier.
↔
Batching to one render per frame adds a flush rule and a pending-frame variable, in exchange for smooth long answers.

Done when you can

I can explain every layer from bytes to pixels in Mitra v2.
I can list the scenarios Mitra v2 handles and how each is shown.
I can explain why the final text is drawn explicitly after the loop.
I can test a parser against emoji, Hindi, tiny packets, empty, cut and length cases.
I can name what Mitra still lacks and the phase that adds it.

Command Palette