PHASE 2Beginner ~25 min· topic 3 of 8

Topic 2.3

Bytes to Text

In one line

The network carries bytes, and some characters take several bytes. A chunk boundary can fall inside a character, so decoding each chunk on its own produces broken text. Use one decoder for the whole stream with stream: true, or TextDecoderStream, which does it for you.

Think of it like this

Mailing a sentence as postcards, where each postcard holds a fixed number of strokes. Simple letters take one stroke. A complicated symbol takes three or four. If a symbol's strokes are split across two postcards, reading each postcard alone gives gibberish, and the symbol only makes sense when you keep the first half until the second arrives. Decoding streamed text is the same job: keep the half-finished character until its other half arrives.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Byte: A number from 0 to 255. The unit the network carries.
UTF-8: The standard way to write text as bytes. A character takes 1 to 4 bytes.
TextDecoder: A built-in object that turns bytes into text.
stream: true: An option on decode() that says more bytes may follow, so an unfinished character is kept for the next call.
TextDecoderStream: A ready-made stream stage that decodes a byte stream into text chunks, handling split characters.
Replacement character (U+FFFD): The � mark a decoder writes in place of bytes it cannot understand.
Surrogate pair: Two UTF-16 units that together make one emoji or rare character. slice can cut between them.
Grapheme: What a person counts as one character on screen. One grapheme can be many code units.

Step by step

01See the sizes

The first example starts by printing how many bytes some characters take. A is 1, é is 2, a Hindi syllable pair is 6, ☕ is 3 and the smiling face is 4. Then it cuts the bytes of चाय ☕ 🙂 at positions 2, 5, 9, 12 and the end, which splits several characters.

02Decode each chunk alone: broken

The example decodes the five chunks one by one with a new decoder each time. The result is full of �, and the line below it reports 6 replacement characters. Nothing threw an error.

wrong.jswhole filejs

// a new decoder for every chunk: a split character cannot be put back together
for await (const chunk of res.body) {
  text += new TextDecoder().decode(chunk);
}

03One decoder, stream: true

Create the decoder once, outside the loop, and pass { stream: true }. The decoder holds back the incomplete bytes. The last two lines of the first example show it exactly: the first two bytes of the smiling face decode to an empty string (held back), and the last two bytes then produce the whole face.

right.jswhole filejs

const decoder = new TextDecoder();           // once, for the whole response
for await (const chunk of res.body) {
  text += decoder.decode(chunk, { stream: true }); // keeps a half character for next time
}
text += decoder.decode();                    // the end: flush anything left

04TextDecoderStream, and what the end of a stream can hide

The second example sends bytes in packets of 3 so characters are cut everywhere, and TextDecoderStream still returns the text intact. Then a stream that stops two bytes into the smiling face ends with �. That one mark, at the end, tells you the connection broke.

It also shows fatal: true throwing TypeError: Failed to execute 'decode' on 'TextDecoder': The encoded data was not valid., and a lone first byte of ☕ decoding to �.

TextDecoderStream, and what the end of a stream can hidediagram

Rendering diagram…

05Counting characters for display

The third example shows that length counts UTF-16 units, so a smiling face has length 2 and a family emoji has length 8, while a person counts 1 for each. slice(0, 2) on text starting a🙂 ends in half an emoji. Intl.Segmenter counts what people see. Use it any time you cut or count answer text for a screen.

Try it yourself

1
Change where the cuts fall
In the first example, change const cuts = [2, 5, 9, 12, bytes.length]; to [4, 8, 12, bytes.length].
With different cut points the careless decode breaks different characters. Try a few sets: the bug depends on where the network cuts, so a real user hits it at random.
2
Make the packets bigger
In the second example, change packets(3) to packets(64). The answer arrives in one packet and the careless code would have worked.
That is why this bug is so easy to miss in development: on a fast local connection most answers arrive in a few big chunks.
3
Throw on bad bytes
In the second example, replace new TextDecoderStream() with new TextDecoderStream("utf-8", { fatal: true }) and run again.
The answer cut inside the emoji now throws Uncaught TypeError: The encoded data is not valid. while it is being read, instead of quietly ending with �. Decide which behaviour you want: a visible error you can handle, or a replacement mark.

Code & diagrams

A character cut in two: wrong and right

The replacement character is printed as �. Nothing throws an error when the decode goes wrong; you only see it in the text.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"a" -> 1 UTF-16 unit(s), 1 byte(s)
"é" -> 1 UTF-16 unit(s), 2 byte(s)
"चा" -> 2 UTF-16 unit(s), 6 byte(s)
"☕" -> 1 UTF-16 unit(s), 3 byte(s)
"🙂" -> 2 UTF-16 unit(s), 4 byte(s)
chunk sizes: 2, 3, 4, 3, 6
decoded chunk by chunk: "����य �� 🙂"
same as the original: false | replacement characters: 6
one decoder, stream: true: "चाय ☕ 🙂"
same as the original: true
first 2 bytes of 🙂 -> ""
last 2 bytes -> "🙂"

TextDecoderStream on a response cut into tiny packets, and one cut short

With packets of 3 bytes, characters are cut in many places, and the text still arrives whole. The unusual � at the end of the second line is the sign that the stream was cut short.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

TextDecoderStream: "Chai ☕ चाय 🙂 done" | intact: true | pieces: 10
ends mid-character: "Chai �" | last char is U+FFFD: true
TypeError: Failed to execute 'decode' on 'TextDecoder': The encoded data was not valid.
lone first byte of ☕: "�"

length, slice and what a person counts

Intl.Segmenter works in all current browsers (MDN lists it as Baseline 2024, available since April 2024). Check a support table before relying on it for older ones.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

smile: length 2 | code points 1 | what a person sees 1
family: length 8 | code points 5 | what a person sees 1
Hindi word: length 3 | what a person sees 2
smile cut at 1: "\ud83d"
slice(0, 2) ends in half an emoji: true
first 3 characters, counted the way a person counts, keep the whole family: true

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Decode every chunk with a new decoder

In the second example, change the first loop to for await (const raw of res.body) {, add const piece = new TextDecoder().decode(raw); as its first line (a new decoder for every chunk), and leave the rest.

terminal

$ The first console line, for the answer cut into packets of 3 bytes

── what you'll see ──

TextDecoderStream: "Chai �� चाय �� done" | intact: false | pieces: 10

Myth vs fact

Myth

A chunk is always valid text.

Fact

A chunk is bytes, and a cut can fall inside a character. Only the whole stream is valid text.

Myth

This only matters for emoji.

Fact

Most Indian-language letters and many symbols take 3 bytes, and accented letters take 2. Any non-English text can be hit, and English with a typographic quote or a ₹ too.

Myth

If decoding fails, I get an error.

Fact

By default the decoder quietly writes �. Only fatal: true throws. Broken text can reach users without any error in your logs.

Myth

string.length is the number of characters.

Fact

It counts UTF-16 units. An emoji can be 2 and a joined family emoji 8. Use Intl.Segmenter when you count or cut text for display.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Models answer in the language the user writes in, and Indian-language text is three bytes per letter, so chunk cuts inside characters are common, not rare. Test with Hindi, Tamil or Bengali and an emoji, not only English, before you call the parser finished.
▸
Add a regression test that feeds your decoding code a response cut into 1-byte packets. If your code is correct at 1 byte per packet, it is correct at any packet size, because every cut position gets exercised.
▸
A final � at the very end of a streamed answer is worth a log line in production. It means the connection ended inside a character, which means the answer was cut off, whatever the status code said.

Remember this

1
The problem this solves: A streamed answer in English looks fine even when decoded carelessly, so the bug hides. Then a user asks in Hindi, or the model adds an emoji, and the page fills with � marks, but only sometimes, depending on where the network happened to cut. This topic explains why and shows the two-line fix.
2
Text is stored as bytes. A byte is a number from 0 to 255. The standard way to write text as bytes is UTF-8. Plain English letters take one byte. Letters with accents take two (é). Most Indian-language letters, and symbols like ☕, take three. Emoji take four. The first live example prints these sizes, so you can see them. A Response body delivers raw bytes, and decoding them back into text is your job.
3
Chunks do not respect characters. The network cuts the byte stream wherever it likes (Topic 2.2). If a cut falls between the first and last byte of one character, the character is split across two chunks. Neither half is a valid character by itself.
4
What a careless decode does. new TextDecoder().decode(chunk) on each chunk separately sees a half character and cannot make sense of it. It substitutes the replacement character U+FFFD, which looks like �, for each bad piece. The real character is gone: the first half became �, and the second half became another �. Nothing throws, so you only find out by looking at the screen.
5
**The fix: one decoder, stream: true. Create one** TextDecoder for the whole response and call decoder.decode(chunk, { stream: true }) for every chunk. The option means "more bytes may follow", so the decoder keeps an unfinished character inside itself and completes it with the next chunk. When the stream has ended, call decoder.decode() once with no arguments to flush whatever is left.
6
Or let the platform do it: res.body.pipeThrough(new TextDecoderStream()) gives you a stream of text strings, with the stream: true bookkeeping done for you. This is what the examples in this course use. The explicit decoder is worth knowing for two reasons: it is what is happening inside, and you need it when you read bytes you got another way.
7
A stream that ends in the middle of a character. If the connection drops halfway through a character, the final flush turns the leftover bytes into one �. That is a real signal that the answer was cut off, not just a cosmetic fault, so a stream that ends this way should be treated as a broken stream (Phase 3). A decoder created with fatal: true throws a TypeError on invalid bytes instead of substituting, which is useful when you want to detect it.
8
Cutting text safely in JavaScript. JavaScript strings count in UTF-16 units, so "🙂".length is 2, and slice can cut an emoji in half. A family emoji made of several joined emoji has length 8 but is one picture to a reader. If you ever truncate an answer (for a preview or a title), count with Intl.Segmenter, which counts the way a person does.

Explain it without notes

Why can decoding each chunk separately produce �?

How does decode(chunk, { stream: true }) fix it?

What does TextDecoderStream do, and when would you use the explicit TextDecoder instead?

A streamed answer ends with a lone �. What does that tell you?

Why is text.slice(0, 100) risky for an answer preview?

Practice

Write decodeAll(chunks) that takes an array of Uint8Arrays and returns the text, safe against split characters. Test it with the bytes of "क्या हाल है 🙂" cut into 1-byte chunks.

Write preview(text, n) that returns the first n characters a person would count, and adds … only when text was cut.

Trade-offs

↔
TextDecoderStream is the least code and the safest default. An explicit TextDecoder gives you access to the moment of flush and to fatal, at the price of remembering the stream: true and final decode() rules.
↔
Replacing bad bytes with � keeps the page alive but hides damage. fatal: true makes damage loud, but an uncaught throw in a stream loop needs handling or the answer vanishes. Many teams keep the default and log when the output ends in �.

Done when you can

I can say how many bytes a letter, an accented letter, a Hindi letter and an emoji take in UTF-8.
I can explain why per-chunk decoding produces � and why English tests miss it.
I use one decoder with stream: true, or TextDecoderStream, for streamed text.
I can tell that a stream ending in � was cut off.
I count and cut display text with Intl.Segmenter, not length and slice.

Command Palette

Words you'll meet

Step by step

01See the sizes

02Decode each chunk alone: broken

03One decoder, stream: true

04TextDecoderStream, and what the end of a stream can hide

05Counting characters for display

Try it yourself

Change where the cuts fall

Make the packets bigger

Throw on bad bytes

Code & diagrams

Break it on purpose

Decode every chunk with a new decoder

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can