PHASE 5Intermediate ~21 min· topic 4 of 8

Topic 5.4

Relaying a Stream to the Browser

In one line

Your server sits between the provider and the browser, so it must pass the stream along as it arrives. Buffering proxies, compression and wrong headers can silently turn a stream into one late lump.

Think of it like this

A chain of people passing buckets of water along a line. If one person waits to collect four buckets before passing any on, the person at the end still gets the same water, but in bursts, long after it started. A buffering proxy is that person.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Relay: Reading a stream from one source and writing it to another as it arrives, without waiting for the end.
Proxy: A server between a client and your server that forwards requests and responses.
Buffering: Holding data back until enough has collected, before sending it on.
Flush: Forcing held data to be sent now.
Reverse proxy: A server in front of your servers (nginx, a load balancer, a CDN) that receives browser requests and passes them on.
Backpressure: A slow reader telling the writer to wait. A relay should not read faster from the provider than the browser can take.
Idle timeout: A limit on how long a connection may stay silent before a proxy closes it.

Step by step

01Step 1: draw the two hops

Browser, your server, provider. The provider streams to your server and your server streams to the browser. A delay at either hop delays the user.

Step 1: draw the two hopsdiagram

Rendering diagram…

02Step 2: read, extract, write

The relay reads the provider's lines, takes out the text piece of each event (using the parser from Topic 5.2) and writes it to the response right away. The three real-code tabs show the same relay as FastAPI, a Next.js route handler and Spring's SseEmitter.

03Step 3: set the response headers

Mark the response as a stream and tell intermediaries not to hold it: a content type, Cache-Control: no-cache and, for nginx, X-Accel-Buffering: no. The headers are free; forgetting them costs hours of debugging in production.

headers.txtwhole filetext

Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache, no-transform
X-Accel-Buffering: no

04Step 4: simulate a buffering proxy

The example sends eight small events and passes them through a pretend proxy that forwards only when its buffer is full or the stream ends. Change the buffer size and watch when the browser gets text and how many times the screen can update.

05Step 5: pass cancel upstream

Tie the two connections together. When the browser disconnects, the response stream's cancel hook (or the request's abort signal) fires: stop reading from the provider. The next topic covers this in detail.

06Step 6: test on the real path

Streaming bugs live in the infrastructure, so test after deploying: curl -N https://your-site/api/chat -d ... (the -N flag turns off curl's own buffering) and watch whether text appears piece by piece. Then test through every proxy or CDN in front of the real site.

Try it yourself

1
Find the threshold
Try buffer sizes 14, 15, 29, 30 and 100. Predict how many screen updates each gives (an event is 14 bytes) before you run. Which smallest size first merges two events?
2
Slow down the server
Change GAP_MS to 400. Does the buffer make the first text later or earlier than before? Compute the first arrival for the 56-byte buffer by hand.
3
Find the buffering in your own stack
List every hop between a browser and your model provider in a project you know (CDN, load balancer, reverse proxy, platform). For each, write down whether it can buffer a response and how to turn that off.

Code & diagrams

A buffering proxy turns a stream into a lump New tab

Eight events are sent 100 ms apart. A pretend proxy forwards them when its buffer is full or the stream ends. Time is simulated.

Expected output

no buffering       arrivals (ms): 500, 600, 700, 800, 900, 1000, 1100, 1200
                   first text after 500 ms, screen updates: 8
buffer 56 bytes    arrivals (ms): 800, 800, 800, 800, 1200, 1200, 1200, 1200
                   first text after 800 ms, screen updates: 2
buffer 4096 bytes  arrivals (ms): 1200, 1200, 1200, 1200, 1200, 1200, 1200, 1200
                   first text after 1200 ms, screen updates: 1

A relay endpoint in three stackspython

Not run here: they need a server framework and an API key. FastAPI, a Next.js route handler (the shape this site's AI tutor uses) and Spring MVC.

# pip install fastapi uvicorn httpx      run with: uvicorn app:app
import json
import os

import httpx
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
URL = "https://api.groq.com/openai/v1/chat/completions"


async def pieces_of_text(prompt):
    """Read the provider's SSE stream and yield only the text, like this site's tutor does."""
    body = {"model": "qwen/qwen3.8-27b", "stream": True, "messages": [{"role": "user", "content": prompt}]}
    headers = {"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]}
    async with httpx.AsyncClient(timeout=None) as client:
        async with client.stream("POST", URL, headers=headers, json=body) as upstream:
            async for line in upstream.aiter_lines():
                if not line.startswith("data:") or line.strip() == "data: [DONE]":
                    continue
                choices = json.loads(line[5:])["choices"]  # empty on a usage-only chunk
                if choices and choices[0]["delta"].get("content"):
                    yield choices[0]["delta"]["content"]
    # leaving the "async with" blocks closes the upstream connection, also when the browser disconnects


@app.post("/chat")
async def chat(prompt: str):
    return StreamingResponse(
        pieces_of_text(prompt),
        media_type="text/plain; charset=utf-8",
        headers={"Cache-Control": "no-cache, no-transform", "X-Accel-Buffering": "no"},  # ask nginx not to buffer
    )

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Just over one event

In the proxy example, change the 56-byte buffer to 15 bytes (one event is 14 bytes).

terminal

$ Run

── what you'll see ──

no buffering arrivals (ms): 500, 600, 700, 800, 900, 1000, 1100, 1200

first text after 500 ms, screen updates: 8

buffer 56 bytes arrivals (ms): 600, 600, 800, 800, 1000, 1000, 1200, 1200

first text after 600 ms, screen updates: 4

buffer 4096 bytes arrivals (ms): 1200, 1200, 1200, 1200, 1200, 1200, 1200, 1200

first text after 1200 ms, screen updates: 1

Myth vs fact

Myth

If streaming works locally, it works in production.

Fact

Locally there is no proxy. In production a CDN, load balancer or platform may buffer, compress or time out the response.

Myth

The browser can call the provider directly to stream.

Fact

That puts your secret key in the browser and removes your limits and logging. Always stream through your server.

Myth

The relay must pass the provider's SSE through unchanged.

Fact

It can reshape it. Sending only plain text pieces keeps the browser simple and provider details private.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Streaming holds one connection open per active reader for the full generation (seconds to a minute). Plan concurrency: file descriptors, worker counts, platform limits on concurrent invocations and on maximum duration.
▸
Send a first byte quickly (headers plus a comment) so proxies and users see that something is happening. Keep-alive comments every 15-30 seconds stop idle timeouts during reasoning pauses.
▸
Errors after the first byte cannot change the HTTP status. Define an in-band way to say "the stream failed" (a final event with an error code) and handle it in the client; do not rely on the status code alone.
▸
Log the relay with the metrics from Topic 5.1: time to first byte from the provider, time to first byte sent to the browser, total time, tokens, and why it ended (finished, client left, provider error).

Remember this

1
Why this exists. The browser should not call the model provider directly: it would need your secret API key (Topic 2.4), and you could not enforce limits or log usage. So the browser calls your server, and your server calls the provider. Now the stream has two hops. Your server must read the provider's stream and write to the browser as each piece arrives, or the streaming gain from Topic 5.1 is lost.
2
What to send to the browser. There are two common choices. You can pass SSE through, or reshape it. This site's AI tutor reads the provider's SSE, pulls out only the text pieces, and sends the browser plain text chunks, so the browser code is a simple loop over a ReadableStream. Reshaping hides provider details, lets you switch provider without changing the browser, and lets you remove hidden reasoning tokens. Passing SSE through is simpler, and the lab on this site does that (it returns the provider's text/event-stream unchanged).
3
Set the headers. For SSE use Content-Type: text/event-stream. Add Cache-Control: no-cache (some setups add no-transform so intermediaries do not rewrite the body). For nginx, X-Accel-Buffering: no asks it not to buffer this response. Avoid response compression for streams, or make sure it flushes: a compressor that waits for a full block holds data back like a buffer.
4
Buffers are the silent killer. Reverse proxies, CDNs, load balancers and some hosting platforms buffer responses by default. With a buffer of a few kilobytes, a reply of 100 tokens may arrive as one lump at the end, or in large steps. The example shows eight events sent 100 ms apart. With no buffer the browser gets 8 updates. With a 56-byte buffer it gets 2. With a 4096-byte buffer all eight arrive together at the end, and the page looks like it did not stream at all. This often works on a developer's laptop (no proxy) and fails after deploying.
5
Pass cancellation and errors along. When the browser closes the connection (the user left or pressed Stop), your server must stop reading from the provider so the provider stops writing. The Next.js example passes request.signal to the provider request and implements cancel() on the stream. The reverse matters too: if the provider fails after you already sent status 200 and some text, you cannot change the status. Signal the failure inside the stream (a special final event) so the UI can show it (Topic 5.7).
6
Platform limits. Serverless platforms limit how long a function may run, and some cap response sizes or buffer differently on free plans. Long generations need a platform and a timeout that allow streaming responses. Keep keep-alive comments (: ping) flowing during long silent pauses, such as a reasoning model thinking, so idle timeouts do not close the connection.

Explain it without notes

Why should the browser stream through your server instead of calling the model provider directly?

List three things that make a stream arrive in lumps after deployment, and how to fix each.

How do you make cancellation travel from the browser to the provider?

What can you not do when the provider fails midway, and what do you do instead?

Practice

Write flush_times(sent_times, buffer_events) that takes the times at which a server sent each event and a buffer size counted in events, and returns when each event arrives at the browser if the proxy forwards a batch only when it holds buffer_events events, or when the stream ends.

Write count_updates(times) that returns how many separate moments the screen can update (the number of distinct arrival times).

Trade-offs

↔
Reshaping the stream into plain text keeps the browser simple but hides finish reasons and usage unless you add them back; passing SSE through keeps every field.
↔
A longer keep-alive interval saves bytes but risks idle timeouts; a shorter one costs almost nothing.
↔
Streaming through your own server gives control and security, and adds a hop that must be kept fast and unbuffered.

Done when you can

Done when you can explain why the browser streams through your server.
Done when you can name the response headers a streaming route needs.
Done when you can find and disable buffering in a proxy, CDN or platform.
Done when you can pass a browser disconnect to the provider request.

Command Palette

Words you'll meet

Step by step

01Step 1: draw the two hops

02Step 2: read, extract, write

03Step 3: set the response headers

04Step 4: simulate a buffering proxy

05Step 5: pass cancel upstream

06Step 6: test on the real path

Try it yourself

Find the threshold

Slow down the server

Find the buffering in your own stack

Code & diagrams

Break it on purpose

Just over one event

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can