The OpenAI-Compatible Stream Format

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Chat Completions

The request and reply layout many providers share: a list of messages in, an assistant message out.

chat.completion.chunk

The object value of each streamed JSON message. One chunk adds a little to the answer.

delta

The change a chunk makes to the answer so far, usually a piece of content.

finish_reason

Why the model stopped: stop, length, content_filter, tool_calls. null until the end.

usage

Token counts for the request: prompt tokens, completion tokens and their total.

[DONE]

The final SSE event, a plain marker and not JSON.

choices

The list of possible answers in a chunk. Chat uses the first one. It can be empty.

role

Who is speaking in a message: system, user or assistant.

1. 1

   Add a keep-alive to a stream

   In the third example, change the start of the first stream from ": keep-alive\n\n" + to ": keep-alive\n\n: still here\n\n" +.

   The first output line does not change. Comments never reach your code, however many a server sends.

2. 2

   Break the JSON

   In the third example, add a stream to the object: "bad JSON": "data: {not json}\n\n".

   The JSON.parse inside readSSE throws, and the example's catch reports bad JSON: "" | finish: (none) | error: Expected property name or '}' in JSON at position 1 (line 1 column 2). A real client must decide what to do with a corrupt event: failing the answer and keeping the text received so far is the safe default.

3. 3

   Count the characters, not the chunks

   In the second example, add a line at the end: serve("/api/chat", fakeOpenAI("😀".repeat(3), { delayMs: 0, firstDelayMs: 0, splitBytes: 1 })); await collect("emoji");

   Even one byte at a time, three emoji arrive as emoji: "😀😀😀" | finish: stop. That is the guarantee TextDecoderStream plus the buffer gives you.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

1. 1

   The problem this solves: Topic 2.4 gave you messages. It did not say what is inside them. Most providers (and many servers you can run yourself) copy one layout, called the OpenAI-compatible Chat Completions format, so one parser serves many models. This topic shows exactly what the messages contain, so that you can read a raw stream in your browser's network panel and write a parser that does not break on the odd ones.

2. 2

   The request. You POST a JSON body with messages (a list of { role, content }, where the role is system, user or assistant), the model name (a placeholder like "provider/model-name" here, because real names change), and stream: true. Providers may need an extra option to include token counts in a streamed reply (OpenAI's is stream_options: { include_usage: true }), so check your provider's documentation. Your own server route sends this to the provider and passes the stream back (Topic 1.5).

3. 3

   **Each event's data is a JSON object, a chat.completion.chunk.** It has an id, a model and an array choices. You read choices[0].delta and choices[0].finish_reason. A delta is the small change this chunk makes to the answer: usually { content: "…" } with the next piece of text. Your job is to add each delta.content to the answer. The first example prints a real stream so you can see every shape.

4. 4

   The first chunk usually carries only the role. It looks like delta: { role: "assistant", content: "" }: no text yet. It announces that the assistant is speaking. Code that does answer += chunk.choices[0].delta.content works on it, because the content is an empty string, but code that expects every chunk to have words will be wrong. Treat a missing, null or empty content as "nothing to add".

5. 5

   The last chunks say why it stopped. Until the end, finish_reason is null. In the final chunk it is a word: stop (the model finished normally), length (it ran out of its allowed length, so the answer is cut off mid-thought), content_filter (the provider blocked it), or tool_calls (the model wants a tool run, Phase 8). The reason decides what the screen must say, which is why Topic 3.2 treats length as its own state.

6. 6

   **Usage and the empty choices.** Many providers also send token counts: usage: { prompt_tokens, completion_tokens, total_tokens }. Some put them in the final chunk. Others send one more chunk after it whose choices is an empty array, so choices[0] is undefined. (OpenAI's documentation describes exactly this for include_usage: one extra chunk before [DONE], with empty choices.) If the stream is cut off early, that usage chunk may never arrive, so usage can be missing even on a stream that showed text. A parser that reads choices[0].delta without checking breaks on a perfectly normal stream, after the answer was complete. Check that choices[0] exists.

7. 7

   **[DONE] is not JSON.** The stream ends with the event data: [DONE]. Those characters are not a JSON document, so JSON.parse on them throws. Compare to the string first. Some servers close the connection without sending it, so a stream that ends with neither a finish_reason nor [DONE] has been cut off, not completed. The fourth stream in the third example is exactly that.

8. 8

   Errors can arrive inside a 200. The HTTP status is sent before the model starts, so a failure that happens later cannot change it. Instead the provider may send an event whose data is { "error": { "message": … } } inside a response that already said 200. Check every parsed chunk for error. Failures before the stream starts do use a real status (400, 401, 429, 500) with a JSON body, which your res.ok check catches (Topic 0.4).

9. 9

   "Compatible" is not "identical". Providers add fields: some stream the model's reasoning in an extra field, some add metadata, some send periodic comment lines. Write the parser to use what it needs and ignore the rest. Never assume a field exists, and never throw on one you do not recognise. One to watch for: a provider may report a refusal in a separate field and leave content empty (check your provider's reference for the name), so a parser that only reads content shows a blank answer. Read your provider's reference for the fields it adds, and treat "no content and no finish reason" as a case to handle, not ignore.

• ↔

  A small hand-written parser is a few dozen lines and teaches the format. A library handles more provider quirks and is tested more, but hides the wire format you need to understand when it breaks.

• ↔

  Yielding typed parts (text, finish, error) keeps the app independent of the provider's format, at the cost of dropping fields you did not model. Keep the raw chunk available in development for debugging.

• I can list the pieces of a streamed answer from role-only chunk to [DONE].

• I treat missing, null and empty content as nothing to add.

• I know what stop, length, content_filter and tool_calls mean for the screen.

• I check that choices[0] exists and look for error in every chunk.

• I can tell a complete stream from a cut-off one using the finish reason and the end marker.