PHASE 11Intermediate ~41 min· topic 4 of 6

Topic 11.4

Speech Input

In one line

Speaking a question needs the user's permission for the microphone, a recorder that starts and stops cleanly, and a way to turn sound into text. Each step can be refused, missing or empty, and the screen must say which one happened and what to do next.

Think of it like this

A drive-through speaker. You press nothing; you talk. If the speaker is off, you hear nothing. If you are too quiet, the person says "sorry, I did not catch that". If you mumble, they read your order back so you can fix it. Voice input needs the same three habits: show that it is listening, say it clearly when it heard nothing, and read the words back for correction.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

getUserMedia: The browser call that asks permission to use the microphone or camera and returns a live stream.
MediaStream: A live feed of sound or video, made of tracks.
Track: One channel of a stream, such as the microphone. Stopping the track releases the device.
MediaRecorder: A browser object that records a stream into chunks of audio or video.
Secure context: A page loaded over https (or from localhost). Powerful features such as the microphone are only offered there.
Speech-to-text: Turning recorded speech into written words with a model.
Interim result: A first guess of the words that may still change while the person is talking. A final result will not change.
SpeechRecognition: The Web Speech API object that listens and returns text. Support and where the audio is processed depend on the browser.
Permissions Policy (allow attribute): Rules that say whether an embedded frame may use features such as the microphone.

Step by step

01Ask, and sort every answer

The first example wraps getUserMedia in openMic(getUserMedia) and runs it against five situations: the user allows, blocks, has no microphone, another app is using it, and the API is missing. The first case uses a stand-in stream. The blocked, missing and busy cases are simulated with the error names the browser uses, because the preview cannot show a real prompt. A last case is the real call in this preview.

Each outcome becomes a state with its own words. "Blocked" tells the person where to change it. "No microphone" does not blame them. "Unsupported" points to typing. The last line is the real call: in this preview it rejects with SecurityError, because the frame is not allowed to use the microphone. The page treats it as its own state, "disabled", with its own sentence.

Ask, and sort every answerdiagram

Rendering diagram…

02Record, stop and release

The second example records a stand-in stream (a silent stream made with the Web Audio API) with a real MediaRecorder. It picks a type with isTypeSupported, shows the recorder state moving from recording to inactive, ignores a second Stop press, and stops the track so the indicator would go off. It also shows that the recorder refuses a type it cannot make.

Length rules use your own timestamps, passed in so the example prints the same every time: under half a second is too short; over a minute is cut.

03Send the audio, get words back

The third example uploads the recording as form data to a route that calls transcribe() with a stand-in model. The route refuses a zero-byte upload with 400 before spending anything. A silent recording returns an empty transcript, and the page shows "I did not hear anything" and does not send. A normal one fills the text box and waits for the person to press send.

app/api/transcribe/route.tswhole filets

import { transcribe } from "ai";

export async function POST(req: Request) {
  const form = await req.formData();
  const audio = form.get("audio");
  if (!(audio instanceof File) || audio.size === 0) {
    return Response.json({ error: "No audio received." }, { status: 400 });
  }
  if (audio.size > 5 * 1024 * 1024) {
    return Response.json({ error: "Recording too long." }, { status: 413 });
  }
  const { text } = await transcribe({
    model: provider.transcriptionModel("your-transcription-model"), // the line that changes in a real app
    audio: new Uint8Array(await audio.arrayBuffer()),
  });
  return Response.json({ text });
}

04The browser's own recogniser

The fourth example is about the button, not the words. In Chromium, calling start() on a recogniser that is already running throws InvalidStateError: that is the double-click bug. The example copies that behaviour in a stand-in, then builds the guarded button: a second press stops, and a browser without SpeechRecognition gets a disabled button with a reason. What your own browser has is printed on the page, because it differs between browsers. Whether results ever arrive also depends on the microphone and the permission, so a real page needs a way out when nothing happens.

05Interim and final words

The fifth example drives a controller with a stand-in recogniser that has the same shape as the real one (start, onresult, onerror, onend) and plays scripted events. A clear sentence shows grey interim text that changes and then turns final. Silence, a blocked microphone, a missing network and a missing device each end in their own message, and a second press while listening is ignored.

The codes no-speech, not-allowed, service-not-allowed, audio-capture, network, language-not-supported and aborted are the error values listed by MDN for SpeechRecognitionErrorEvent.

Try it yourself

1
Block the microphone
In the first example change the first case to ["user blocks", reject("NotAllowedError", "Permission denied")] and look at the message.
It tells the person where to change the setting, because the browser will not show the prompt again once the site is blocked. A message that only says "error" leaves them stuck.
2
Change the length rules
In the second example change MIN_MS to 3000 and read the three verdicts.
A 2-second question is now "too short". Short voice questions are normal, so this limit is wrong. Choose limits from how long real questions are, and test them with real people.
3
Send the transcript automatically
The third example only fills the box. Imagine its Send button had a handler that posts the box to the chat, and dictate called send.click() right after filling it.
The noisy recording ("wher iz mai ordur") would be sent as it is, before the person has read it, and the assistant would answer a question nobody asked. That is why the words wait in the box. Topic 11.6 breaks this on purpose with the real chat.

Code & diagrams

Ask for the microphone: every outcome

The blocked, missing and busy cases are simulated with the error names from MDN. The last line is the real call in this preview: the frame is not allowed to use the microphone, so it rejects with SecurityError. In your own page, with permission, that call opens the browser's prompt.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

user allows -> recording: "Listening…"
  track before stop: live
  track after stop: ended
user blocks -> blocked (NotAllowedError): "The microphone is blocked. Allow it in the site settings (the icon next to the address), then press the button again."
no microphone -> nomic (NotFoundError): "No microphone found. Plug one in, or type your question instead."
another app has it -> busy (NotReadableError): "The microphone is in use by another app. Close it and try again."
something unexpected -> failed (AbortError): "The microphone could not start. Try again, or type your question."
no mediaDevices at all -> unsupported: "Voice input is not available here. You can type your question."
this preview's real microphone -> disabled (SecurityError)

Record, stop, and refuse bad recordings

A real MediaRecorder on a silent stand-in stream. Which types a browser can record differs, so the code asks with isTypeSupported and never assumes one.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

a supported recording type exists: true
recorder state: recording
recorder state: inactive
got an audio blob: true
second Stop press: null
120 ms -> too short: hold the button while you speak
2000 ms -> ok
61000 ms -> too long: only the first 60 seconds are used
audio/ogg: NotSupportedError -> use a type from isTypeSupported
track after stop: ended

Send the audio, get words back, handle silence

The route refuses zero bytes and oversized uploads before calling the model. Silence is a valid answer from the model with no words, and gets its own message. The noisy case shows why the words go into the box and are not sent on their own.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

status: Check the words, then press Send. | box: "Where is my order ORD-1042?" | send enabled: true
status: Check the words, then press Send. | box: "wher iz mai ordur" | send enabled: true
status: I did not hear anything. Check your microphone and speak a little closer. | box: "" | send enabled: false
status: The recording could not be used. Try again. | box: "" | send enabled: false
status: That recording is too long. Try a shorter question. | box: "" | send enabled: false

Guard a double press and a missing API

Chromium throws InvalidStateError when start() is called while the recogniser is running, and the stand-in copies that. The first line of the page says what your own browser has. Always guard start() so a double click does nothing.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

unguarded second press: InvalidStateError
presses: listening, stopped, listening, stopped
no API: Dictate (not available here, type instead), disabled: true

Interim words, final words and every error

Scripted events with the same shape as the real API. The first scenario shows interim words turning final. Every error code gets a sentence, and a code the page has not seen gets a safe default.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

== a clear question
listening: ""
ignored: "already listening"
listening: "where is" (still guessing)
listening: "where is my order" (still guessing)
listening: "Where is my order?" (final)
done: "Where is my order?"
== silence
listening: ""
error: "I did not catch anything. Try again, a little closer to the microphone."
== blocked
listening: ""
error: "The microphone or speech service is blocked. Allow it in the site settings."
== offline
listening: ""
error: "Voice input needs an internet connection. Try again or type instead."
== no microphone
listening: ""
error: "No microphone was found. Plug one in or type instead."
== an unknown code
listening: ""
error: "Something went wrong with voice input."

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget to stop the track

In the first example, delete the line r.stream.getTracks().forEach((t) => t.stop()); // releases the device and turns the indicator off and run it.

terminal

$ Console, the two track lines

── what you'll see ──

track before stop: live

track after stop: live

Myth vs fact

Myth

If getUserMedia fails, the user said no.

Fact

It also fails when there is no microphone (NotFoundError), when another app holds it (NotReadableError), and when the page is not allowed to ask. Each needs a different message.

Myth

Voice input works in every browser the same way.

Fact

MediaRecorder formats differ, and SpeechRecognition is not available everywhere and on some browsers sends the audio to a server (MDN). Detect, and keep typing.

Myth

An audio Blob with size greater than zero contains speech.

Fact

Even a recording of silence or of zero length has container bytes. Measure the duration yourself and check the transcript.

Myth

Send the transcript straight to the chat.

Fact

Recognition makes mistakes. Show the words in the box and let the person correct them.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Show that you are listening in more than colour: the words "Listening…", an animated level meter or pulse, and a button whose label changes to "Stop". Announce changes through a polite live region (Topic 10.6), but not every interim word.
▸
Audio is personal data. Tell people before the first use if the audio leaves their device, which is the case for server-side transcription and for browsers whose recogniser is cloud-based. Do not log or keep recordings longer than needed (Topic 9.7).
▸
Hold-to-talk versus click-to-toggle: hold-to-talk needs pointer events with setPointerCapture, and it must also work from the keyboard (Space held) and must cope with the finger sliding off the button. Many teams ship click-to-toggle first.
▸
Test the unhappy paths on real devices: a Bluetooth headset that connects mid-recording, a laptop lid closing, a call arriving on a phone, a permission revoked in settings while the page is open. Browsers usually end the stream's track in these cases, so listen for the track's ended event and show the idle state with a message (how each browser behaves differs, so test the ones you support).

Remember this

1
The problem this solves: Typing is hard on a phone, while walking, or for someone with a hand or eye condition, and speaking is natural. But the microphone is the most sensitive thing a web page can ask for. The browser asks the person first, the person can say no, a computer may have no microphone, and the sound then has to become text. Every one of those steps can fail, and a silent button that does nothing is the worst result.
2
Step 1: permission. navigator.mediaDevices.getUserMedia({ audio: true }) asks the browser to open the microphone. It returns a promise of a MediaStream (the live sound). If the user agrees, the browser shows a recording indicator. The promise rejects with a named error (MDN): NotAllowedError (the user said no, the site is blocked, or the page is not allowed to ask), NotFoundError (no microphone), NotReadableError (the microphone exists but the system or another app prevents its use) and, for an unusable constraint, OverconstrainedError. Turn each into its own sentence.
3
Where it is not available at all. navigator.mediaDevices exists only in a secure context (an https page, or localhost). On an insecure page it is undefined and calling getUserMedia throws a TypeError. A page inside an <iframe> also needs the frame's allow="microphone" attribute, or the browser never even asks. The preview you are using is such a frame without that permission: it is a secure context, navigator.mediaDevices exists, and getUserMedia rejects with SecurityError (MDN: user media is disabled for the document). So the blocked path is what you really see here, and the working path is shown as code and with a stand-in stream.
4
Step 2: record. new MediaRecorder(stream, { mimeType }) records the stream. It fires dataavailable with a Blob of the sound, and stop when finished. Which formats a browser can record differs, so ask with MediaRecorder.isTypeSupported(type) and pick the first that works; never hard-code one. The recorder can also refuse a type with NotSupportedError. When you finish, call stop() on every track of the stream (stream.getTracks().forEach((t) => t.stop())) or the browser's recording indicator stays on.
5
Step 3: words. There are two ways to get text from speech. (A) Send the recorded audio to your server and a speech-to-text model (transcribe() in the AI SDK). You control the model and the privacy, you wait until the recording ends, and the audio is uploaded. (B) The browser's Web Speech API, SpeechRecognition: it gives words while the person is still talking (interim results). MDN says that on some browsers, like Chrome, this uses a server-based service: the audio leaves the device and it does not work offline. It is not available in every browser, so feature-detect and keep typing as the fallback.
6
Empty and bad audio. A recording can be zero seconds long, or silent, or only noise. Silence is not an error: the model returns an empty transcript. Treat it as its own state ("I did not hear anything") with a hint (check the microphone, speak closer), and do not send an empty question to the chat. Also set a minimum and a maximum length yourself, measured by your own clock: in Chromium a Blob from MediaRecorder is never really empty, because even an instant recording has container bytes.
7
Never send the words without a look. Speech recognition makes mistakes, especially with names, order numbers and accents. Put the transcript into the text box and let the person edit it before sending. That is the voice version of reading the order back. Mark the state clearly: not listening, listening, working on it, done, or the reason it did not work.

Explain it without notes

What can go wrong between pressing a microphone button and getting a transcript?

How do NotAllowedError, NotFoundError and NotReadableError differ, and what does the page say for each?

What are the two ways to turn speech into text on a web page, and what does each cost?

How should the UI treat a silent recording?

Why release the stream's tracks, and how?

Practice

Add an OverconstrainedError case to the first example and decide on the sentence. Then write describeMic(state) that returns { icon, label } for each state for a button.

Write a useMicPermission() hook outline that checks navigator.permissions.query({ name: "microphone" }) inside a try, and returns "unknown" when the browser does not support it.

Make the controller in the last example call recognition.stop() on the second press and finish with what has been heard so far.

Trade-offs

↔
Server-side transcription is consistent across browsers and under your control, but it uploads audio, costs per minute and answers only after the recording. Browser recognition is instant and free to run, but varies by browser and may send audio to a third party.
↔
Click-to-toggle is easy and accessible; hold-to-talk is quick and forgiving about length but harder to build well. A mic button that does something clear when it fails matters more than either.

Done when you can

I can ask for the microphone and tell the five outcomes apart.
I can record with MediaRecorder and choose a supported type.
I can release the microphone when recording ends.
I can show silence, noise and a failed upload as different states.
I can show interim and final words and put them in the box for correction.
I can say what leaves the device in each approach.

Command Palette

Words you'll meet

Step by step

01Ask, and sort every answer

02Record, stop and release

03Send the audio, get words back

04The browser's own recogniser

05Interim and final words

Try it yourself

Block the microphone

Change the length rules

Send the transcript automatically

Code & diagrams

Break it on purpose

Forget to stop the track

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can