PHASE 11Intermediate ~39 min· topic 5 of 6

Topic 11.5

Speech Output

In one line

To read an answer aloud while it is still arriving, cut the stream into whole sentences, clean the markup out, and queue each sentence for a voice. The page must stop the voice at once when the user asks, ignore the noise of its own cancelling, and cope with browsers that will not play sound until the user taps.

Think of it like this

A radio newsreader reading from a printer that is still printing. She cannot start a sentence until she has the whole sentence, or she might stop in the middle of a word. She skips the page numbers and the symbols. And when the producer shouts "stop", she stops at once without arguing. Reading a streamed answer aloud is the same job: whole sentences, no markup, and an instant stop.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

speechSynthesis: The browser's text-to-speech controller: speak, cancel, pause, resume, getVoices.
Utterance: One piece of text to be spoken, as a SpeechSynthesisUtterance with optional voice, rate and pitch.
Voice: One of the speakers the device offers. The list differs by device and may load late.
Intl.Segmenter: A built-in that splits text into graphemes, words or sentences using the rules of a language.
Chunking: Cutting a stream of text into pieces that are good to speak, here whole sentences.
Interrupted / canceled: The error codes for an utterance stopped while speaking / while still queued. Normal after a cancel.
Autoplay policy: The browser's rule that blocks sound a page starts without a user action.
NotAllowedError: What audio.play() rejects with when the browser refuses to start sound.
Text-to-speech (TTS): Turning written text into spoken audio.

Step by step

01Where sentences end

The first example splits sentences three ways: a naive split on full stops, Intl.Segmenter, and Intl.Segmenter plus the minimum-length rule. Read the first case: both of the first two cut Dr. off from the name. The third keeps them together. Hindi and Japanese split on their own marks, and an emoji, a URL with dots, a number such as 3.5 and a text with no full stop at all are included.

02Speak as it streams

The second example feeds a streamed answer into a small speaker. After each token it asks: is there a complete sentence yet? If so it cleans it and queues it. Each "queue" line in the output happens at a stream event, and the stand-in voice finishes utterances only when the script says so, which makes the order repeatable.

In a real page the browser's voice takes real time for each sentence, and the queue builds up behind it. That is the normal case: the text arrives faster than speech.

Speak as it streamsdiagram

Rendering diagram…

03Stop, and ignore your own noise

In the third example the user presses Stop after the second sentence is queued. The speaker cancels the voice, empties its buffer, and ignores the tokens still arriving. The first sentence reports interrupted, the second canceled. The page treats both as normal and shows nothing. Then the example feeds the error codes MDN lists through the same handler to show which ones deserve a message.

04The real voice in this preview

The fourth example calls the real speechSynthesis and waits for the utterance to end or fail. What happens depends on the device: it may speak, or it may end in an error event such as not-allowed (in the checker's browser it does, and nothing is spoken). The result is written on the page, and the console only says that the voice answered. Whatever happens, the page must have an answer for it, and a time-out for the case where no event ever arrives.

05A server voice and the autoplay block

The fifth example makes a route that calls generateSpeech with a stand-in model that returns a tiny silent WAV file. The page turns the response into an object URL and calls play() from script, with no user action. The browser rejects with NotAllowedError, the page shows a "Listen" button instead, and the output also prints navigator.userActivation.hasBeenActive, which is false until the user has pressed something. The route also shows an empty text (400), a too-long text (413) and a busy service (429).

app/api/speak/route.tswhole filets

import { generateSpeech } from "ai";

export async function POST(req: Request) {
  const { text } = await req.json();
  if (typeof text !== "string" || !text.trim()) return Response.json({ error: "Nothing to say." }, { status: 400 });
  if (text.length > 1000) return Response.json({ error: "Too long." }, { status: 413 });
  const { audio } = await generateSpeech({
    model: provider.speechModel("your-speech-model"), // the line that changes in a real app
    text,
    outputFormat: "mp3",
  });
  return new Response(audio.uint8Array, { headers: { "content-type": audio.mediaType } });
}

Try it yourself

1
Change the minimum length
In the second example change min = 24 to min = 60 and run it.
The first two sentences are now queued as one piece (Your order ORD-1042 shipped on Monday. It should arrive by Friday!), because the first is shorter than 60 characters. A high minimum gives smooth long pieces and a later start for each one. Go the other way (min = 5) and nothing changes here, because every sentence of this answer is already longer than 5 characters; the number only matters for fragments such as Dr.. Choose it by listening, with your own text.
2
Speak the link as it is
In cleanForSpeech delete the line that replaces the bare address with "link" and the line that turns Markdown links into text, then run the second example.
The queue line now reads Track it here: [track my order](https://chai.example/track/1042)., with the whole address inside. A voice would read the brackets and spell the address out, which is a long and useless noise.
3
Show every error
In the third example make onSpeechError return a message for interrupted and canceled too.
Pressing Stop now shows two error messages for something the user asked for. Cancelling is not a failure.

Code & diagrams

Where do sentences end?

Both the naive split and Intl.Segmenter cut after abbreviations such as Dr. and Rs. in this browser. The minimum length rule is what keeps them together. The numbers 24 and 8 are choices: tune them to your voice and your language.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"Dr. Rao said the tea weighs 3.5 kg. Your order shipped!"
  naive:    ["Dr.","Rao said the tea weighs 3.5 kg.","Your order shipped!"]
  segmenter: ["Dr.","Rao said the tea weighs 3.5 kg.","Your order shipped!"]
  merged:    ["Dr. Rao said the tea weighs 3.5 kg.","Your order shipped!"]
"It costs Rs. 250. Fine."
  naive:    ["It costs Rs.","250.","Fine."]
  segmenter: ["It costs Rs.","250.","Fine."]
  merged:    ["It costs Rs. 250. Fine."]
"Your order is on its way 🎉 It arrives on Friday. Enjoy"
  naive:    ["Your order is on its way 🎉 It arrives on Friday.","Enjoy"]
  segmenter: ["Your order is on its way 🎉 It arrives on Friday.","Enjoy"]
  merged:    ["Your order is on its way 🎉 It arrives on Friday.","Enjoy"]
"Tracking: https://chai.example/track/1042. Thanks for waiting."
  naive:    ["Tracking: https://chai.example/track/1042.","Thanks for waiting."]
  segmenter: ["Tracking: https://chai.example/track/1042.","Thanks for waiting."]
  merged:    ["Tracking: https://chai.example/track/1042.","Thanks for waiting."]
"आपका ऑर्डर भेज दिया गया है। यह शुक्रवार को पहुंचेगा।"
  segmenter: ["आपका ऑर्डर भेज दिया गया है।","यह शुक्रवार को पहुंचेगा।"]
  merged:    ["आपका ऑर्डर भेज दिया गया है।","यह शुक्रवार को पहुंचेगा।"]
"注文は発送されました。金曜日に届きます。"
  segmenter: ["注文は発送されました。","金曜日に届きます。"]
  merged:    ["注文は発送されました。","金曜日に届きます。"]
"Hello"
  naive:    ["Hello"]
  segmenter: ["Hello"]
  merged:    ["Hello"]
""
  naive:    []
  segmenter: []
  merged:    []

Speak each sentence as soon as it is complete

The 'queue' lines happen as the text streams in. The stand-in voice only finishes when asked, so you can read the order. Real voices speak for real time while later sentences wait in the queue.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

queue: "Your order ORD-1042 shipped on Monday."
  start: Your order ORD-1042 shipped on Monday.
queue: "It should arrive by Friday!"
queue: "Track it here: track my order."
queue: "Anything else?"
-- the voice works through the queue
  end:   Your order ORD-1042 shipped on Monday.
  start: It should arrive by Friday!
  end:   It should arrive by Friday!
  start: Track it here: track my order.
  end:   Track it here: track my order.
  start: Anything else?
  end:   Anything else?

Stop: interrupted and canceled are not errors

Stop is pressed from the stream: right after the second sentence is queued. The first sentence was already speaking, so it is interrupted. The second was waiting, so it is canceled. Neither shows a message.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

queued 1: First sentence is short enough.
queued 2: The second one is long enough too.
  interrupted: "First sentence is short enough." -> message shown: null
  canceled: "The second one is long enough too." -> message shown: null
stopped: buffer cleared, voice cancelled
stop pressed again: nothing to do
tokens that arrived after stop were dropped: true
nothing left in the voice queue: true
-- other error codes
not-allowed -> Tap the Listen button to allow sound.
synthesis-unavailable -> This device has no voice installed. You can read the answer instead.
language-unavailable -> No voice for this language. You can read the answer instead.
text-too-long -> That passage is too long to read aloud.
synthesis-failed -> Reading aloud failed. You can read the answer instead.

The real browser voice in this preview

The real voice depends on the device, so the result is written on the page and the console only says whether the voice answered. The checker's browser answers with an error event (not-allowed) and nothing is spoken; yours may speak. The time-out is there only so this example cannot hang.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

speechSynthesis available: true
the voice answered with an end or an error event: true

A server voice, and the autoplay block

Nobody has pressed anything in this page, so the browser refuses to start sound. When the user presses a real button, play() inside that click is allowed. getAutoplayPolicy is not in every browser, so the example feature-detects it and shows the result on the page.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

empty text -> 400 Nothing to say.
too long -> 413 Too long.
busy service -> 429 The voice is busy.
audio received: 200 audio/wav, has bytes: true
user has interacted with the page: false
AudioContext starts as: suspended
play() rejected: NotAllowedError -> show a Listen button, and call play() inside its click handler

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Leave the last piece for later, forever

In the second example delete the line speaker.end(); // the stream is over: speak what is left, even without a full stop and run it.

terminal

$ Console, the queue lines

── what you'll see ──

queue: "Your order ORD-1042 shipped on Monday."

queue: "It should arrive by Friday!"

queue: "Track it here: track my order."

(no line for "Anything else?": it is never spoken)

Myth vs fact

Myth

Send each token to the voice as it arrives.

Fact

A voice needs whole sentences. Token by token it stutters, mispronounces half-words and has no pauses.

Myth

Intl.Segmenter knows abbreviations.

Fact

In the browser used here it cut after Dr. and Rs.. Add a minimum length, and test with your own text and languages.

Myth

An error event from the voice means something broke.

Fact

interrupted and canceled are what cancel() produces. Only the other codes are problems.

Myth

Once a page works on your laptop, speech works everywhere.

Fact

Voices differ per device, may load late, may be missing, and sound is blocked until the user has interacted with the page.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Wait for voices. speechSynthesis.getVoices() can return an empty list at first and fill later (MDN: on some browsers such as Chrome, wait for voiceschanged). Choose a voice by language (voice.lang) after that event, and fall back to the default voice if none matches.
▸
Set utterance.lang to the language of the text, not of the interface. A Hindi answer read by an English voice is unintelligible. If you do not know the language, let the model say it in structured output (Topic 7.3) or skip speaking.
▸
Never read secrets, tool arguments or the model's reasoning aloud. Speech is public in a room. Speak the final answer text only, and give the user a mute control that is always visible.
▸
Pause when the tab hides, and when the user starts to type or talk: speaking over a person who is dictating is rude and ruins the recognition (Topic 11.4). Cancel the voice before opening the microphone.

Remember this

1
The problem this solves: A model's answer arrives one small piece at a time, and a voice speaks best in whole sentences. If you send every piece to the voice, it stutters and reads half-words. If you wait for the whole answer, the listener waits as long as the reader would have. The craft is to speak each sentence as soon as it is complete, and to keep the voice and the text on the screen under the same control.
2
There are two voices. (1) The browser's own: speechSynthesis.speak(new SpeechSynthesisUtterance(text)). It is free, works offline with local voices, and sounds different on every device. The voice list can arrive late: MDN says that on some browsers, such as Chrome, you must wait for the voiceschanged event before getVoices() is filled. (2) A server voice: your route calls generateSpeech({ model, text }) and returns an audio file (MP3 or WAV) that the page plays with an Audio object. It sounds the same for everyone, costs per character, and has a short delay per sentence.
3
Cut at sentence ends. Intl.Segmenter with granularity: "sentence" splits text by the rules of a language, including Hindi's । and Japanese 。. It is not perfect: in the browser used here it treated Dr. and Rs. as sentence ends. The fix is a minimum length: collect segments until the text has at least, say, 24 characters before you speak it, so Dr. Rao said… stays together. Count characters in a way that fits the language (a Japanese sentence of 8 characters is a full sentence).
4
The last segment may still be growing. While text streams, the final segment might be half a sentence. Treat it as complete only when it ends with ., ! or ? followed by a space (the next sentence has begun), or when the stream is over. At the end, speak whatever is left even if it is short or has no full stop.
5
Clean what is spoken, not what is shown. A voice reads ** as "star star" and a link as a long address. Before speaking, remove Markdown marks, turn [text](url) into text, and replace a bare address with the word "link". Do not read code aloud; say there is a code example on screen (Topic 6.2 covers fenced blocks).
6
Stopping. speechSynthesis.cancel() empties the queue and stops the current utterance. Each cancelled utterance fires an error event: interrupted if it had started speaking, canceled if it was still waiting (MDN's definitions). These are not failures and must not show an error message. Real failures have other codes: not-allowed (the start was not allowed), synthesis-unavailable, language-unavailable, voice-unavailable, text-too-long, audio-busy and more. Tie one stop control to both the stream (Topic 3.1) and the voice: a Stop that silences the text but not the voice feels broken.
7
Autoplay rules. Browsers block sound that the page starts on its own. MDN: playback of media with audio is generally blocked when it is started by script in a tab where the user has not yet interacted with the page; play() then rejects with NotAllowedError. A Web Audio AudioContext starts suspended in the same situation. The answer is a button the user presses ("Listen"): playing inside that click is allowed, and from then on the page can speak the answer as it arrives.

Explain it without notes

Why speak sentences and not tokens, and how do you find sentence ends in a stream?

What do interrupted and canceled mean, and how should a page treat them?

Why does audio.play() sometimes reject, and how do you design for it?

Compare the browser's voice with a server-generated voice.

What must you remove from a Markdown answer before you speak it?

Practice

Remove citation markers like [1] and [2] in cleanForSpeech, and test with "Returns are accepted within 7 days [1]."

Make the speaker skip fenced code blocks and say "There is a code example on screen" once.

Write pickVoice(voices, lang) that prefers an exact language match, then the same language with a different region, then the default voice.

Trade-offs

↔
Speaking sentence by sentence gives a fast start, with a pause at each seam. Waiting for the full answer sounds smoother but starts late. A minimum length of a sentence or two is the usual middle.
↔
Reading aloud by default surprises people in public places. Make it opt-in with a visible control, and remember the choice.

Done when you can

I can cut a streaming answer into sentences, including the last one.
I can clean Markdown and links before they are spoken.
I can stop the voice and the stream together without showing false errors.
I can tell interrupted and canceled from real speech errors.
I can explain why play() is rejected and design a Listen button for it.
I can choose between the browser voice and a server voice.

Command Palette

Words you'll meet

Step by step

01Where sentences end

02Speak as it streams

03Stop, and ignore your own noise

04The real voice in this preview

05A server voice and the autoplay block

Try it yourself

Change the minimum length

Speak the link as it is

Show every error

Code & diagrams

Break it on purpose

Leave the last piece for later, forever

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can