PHASE 11Intermediate ~41 min· topic 6 of 6

Topic 11.6

Build: Mitra Sees and Listens

In one line

Mitra gets two new inputs. A customer can attach a screenshot of a problem (checked, shrunk and previewed before it is sent) and can dictate a question with their voice (the words go into the box for correction). Every refusal has its own message.

Think of it like this

The shop's counter gets a new window and a new bell. Through the window, customers can hold up a photo of the problem. At the bell, they can say what they need. The staff look at the photo before accepting it, and repeat the spoken order back. Mitra's new inputs work the same way: look first, repeat back, then act.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Composer: The input area of the chat: text box, attach, dictate and send controls.
Attachment chip: A small preview with a name and a remove button for a file that is chosen but not yet sent.
File part: A message part { type: "file", mediaType, filename, url } sent with the text.
Interim line: The grey line that shows words the recogniser is still guessing.
Stand-in recogniser: A scripted object with the same shape as SpeechRecognition, used because the preview has no microphone.
Refusal message: A sentence that says what is wrong and what to do, instead of a status code.

Step by step

01Start from Mitra

Mitra is the useChat app from Phase 5, with the Phase 10 polish. The route streams a scripted stand-in model, and instructions carries Mitra's job (in AI SDK 7 the option is instructions; system still works but is deprecated). The model in this phase cannot see images: it is scripted to say that an image arrived. With a real vision model, the only change is the model on the route.

02Step 1: prepare an attachment

The first example is the helper module on its own: validate(file) rules (empty, type, count) and prepare(file), which decodes, shrinks and re-encodes. It runs eight files through them: a large PNG, a transparent PNG, an empty file, an executable, a cut-off image and others. Everything the composer needs to say comes from the results.

attachments.jswhole filejs

export function validate(file, countSoFar) {
  if (countSoFar >= MAX_FILES) return "You can attach up to " + MAX_FILES + " images.";
  if (file.size === 0) return file.name + " is empty.";
  if (!ALLOWED.includes(file.type)) return file.name + " is not a PNG, JPEG or WebP image.";
  return null;
}

03Step 2: the Composer

Add the controls: a file input behind a paperclip button, a drop zone around the form, a paste handler on the box, a list of chips, a microphone button. All three file paths call addFiles, which validates each file, prepares it, and adds a chip, or sets #error with the reason. Setting input.value = "" afterwards lets the same file be chosen again.

04Step 3: send

send converts each prepared JPEG to a data URL (a file part) and calls sendMessage({ text, files }). It then revokes the preview URLs and clears the chips. If the route refuses, useChat reports the error, and the page shows the route's sentence and keeps the text the person typed.

05Step 4: dictate

The microphone button creates the recogniser on first use. A listening state blocks a second start(). Interim words go to #interim, final words are appended to the box, and the button label changes between "Dictate" and "Stop dictating". With no API the button is disabled and carries a title that says why.

Step 4: dictatediagram

Rendering diagram…

06Watch it run

The final example is a scripted visitor. It dictates "Where is my order ORD-1042?", drops in a 2400 by 1600 screenshot, sends, reads the answer, then tries an executable, then four images at once, then an image the route refuses. The console prints what the page shows after each step.

Try it yourself

1
Turn voice off
In the last example change <Composer makeRecognition={standIn(dictation)} /> to <Composer />.
The Dictate button is disabled and its title explains why. The typing path is untouched. The scripted visitor clicks a disabled button, so nothing happens and its first wait never finishes: the console stays empty. That is the point: the page still works, and only the voice feature depends on the API.
2
Lower the image limit
Change MAX_DATA_URL = 30000 to MAX_DATA_URL = 3000 and run again.
Even the shrunk screenshot is now refused by the route with a 413: the page shows Image too large. in the red line and the status becomes error. The scripted visitor waits for an answer that never comes, so only its first two console lines print. The client shrinks to a size you chose; the server enforces the limit you set. They have to agree.
3
Let one more image in
Change const MAX_FILES = 3; to const MAX_FILES = 5; in both the route and the client part (it is one constant).
The four-image drop is now accepted in full, but the scripted visitor then crashes with Cannot read properties of null (reading 'textContent'), because there is no #error element when nothing went wrong. Change that line to (($("#error") || {}).textContent || "(none)") and it prints after four images, chips: 4, error: (none). Providers limit images per message too, so check your provider's current limit before you raise yours.
4
Block the microphone
Change the dictation line to const dictation = [{ error: "not-allowed" }]; and, in the visitor, replace the wait for ORD-1042 with await sleep(200); followed by console.log($("#error").textContent, $("#dictate").textContent, JSON.stringify($("#prompt").value));.
The page shows The microphone is blocked. Allow it in the site settings., the button goes back to Dictate, and the box stays empty (""). A failed dictation never leaves the composer stuck in a listening state, and typing still works.

Code & diagrams

Step 1: validate and prepare an image

Mitra's attachment rules without any screen. The fourth good image is refused only because three are already accepted. Reasons are sentences the composer can show as they are.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

screenshot.png: ok -> screenshot.jpg 1024x683 image/jpeg
empty.png: refused -> empty.png is empty.
setup.exe: refused -> setup.exe is not a PNG, JPEG or WebP image.
notes.txt: refused -> notes.txt is not a PNG, JPEG or WebP image.
cut-off.png: refused -> cut-off.png could not be read. Try another copy.
small.png: ok -> small.jpg 300x200 image/jpeg
logo.png: ok -> logo.jpg 64x64 image/jpeg
fourth.png: refused -> You can attach up to 3 images.

Mitra Sees and Listens

Mitra after Phase 11. The stand-in recogniser plays two scripted events, because the preview has no microphone. The stand-in model cannot see the image: it only reports that one arrived. Everything else (checks, shrinking, previews, errors) is the real browser code.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

dictated into the box: "Where is my order ORD-1042?" (not sent yet, messages: 0)
chips: screenshot.jpg (1024x683) x
you: Where is my order ORD-1042? + 1 image
Mitra: I received 1 image(s). This stand-in cannot look inside them. Order ORD-1042 (Masala chai 250 g) has shipped.
error: setup.exe is not a PNG, JPEG or WebP image.
after four images, chips: 3, error: You can attach up to 3 images.
route said: Image too large. (chat status: error)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Send the dictation automatically

In r.onend, replace the line if (finals) setText((t) => (t ? t + " " : "") + finals); // into the box, not sent with if (finals) sendMessage({ text: finals });. Press Dictate on the page and watch the message list.

terminal

$ The message list on the page, with no one pressing Send (the visitor's wait for the box never ends, so read the page or log the .msg elements)

── what you'll see ──

user: Where is my order ORD-1042?

assistant: Order ORD-1042 (Masala chai 250 g) has shipped.

Myth vs fact

Myth

Voice and images are two separate features to bolt on.

Fact

They share one composer, one send path and one error line. Building them separately gives two different error styles on one screen.

Myth

The route can trust the composer's checks.

Fact

The route re-checks count, type and size. The composer's checks only save a round trip.

Myth

Dictation should send when the person stops talking.

Fact

Silence also ends when the user thinks. Recognition errors need a human look.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep previews as object URLs and the sent copy as a data URL only at send time. Holding the base64 text of three images in React state makes every keystroke slow.
▸
Log the right things. Record that an image was sent, its size and type, and the outcome; do not log the image or the audio (Topic 9.7).
▸
Keep attachments out of saved chats. A conversation in localStorage with data URLs fills the quota fast (Topic 4.4). Store a note such as '2 images were attached' and drop the bytes, or upload and store links.
▸
Accessibility: every chip has a name and a labelled remove button, the previews have empty alt (the name beside them is the label), the sent images have alt with the file name, and errors use role="alert".

Remember this

1
The problem this solves: Topics 11.1 to 11.5 built the pieces separately: file checks, resizing, microphone states, recognition. A real feature needs them in one composer, with the chat from Phase 5 around it, and with every refusal reaching the screen in words. This topic wires them into Mitra, the *Chai & Co* assistant, and says what is still missing.
2
Where Mitra starts. After Phase 10, Mitra is a useChat app with a route at /api/chat (streamText, convertToModelMessages, toUIMessageStreamResponse), a MessageList that renders text parts, and a Composer with #prompt, #send and #stop, starter prompts, feedback buttons and accessible live regions. The example here keeps the parts that this phase touches: Message, Composer and the route.
3
What the Composer gains. A paperclip control (#attach, an <input type="file" multiple>), a drop and a paste handler, a list of chosen images with remove buttons (#files), and a microphone button (#dictate). All attachment paths call one function, addFiles, which validates (empty, type, count), shrinks the image to at most 1024 pixels on its longest side, exports JPEG with a white background, and shows a preview. Previews are object URLs, revoked when removed or sent.
4
What the route gains. It checks the file parts itself: at most three, only PNG, JPEG or WebP, and a size limit on each data URL. The client check is a kindness; the route check is the rule (Topic 11.1). A refusal returns a JSON error with a proper status (400, 413, 415), and the page turns it into a sentence.
5
Voice. The microphone button exists only when the browser has SpeechRecognition. It toggles: press to listen, press again to stop; a second start() is never called. Interim words appear in a line above the box (#interim), and final words are appended to the text already typed. They are not sent: the person reads them, corrects them and presses Send. Errors (no-speech, not-allowed, network…) get sentences. Without the API, the button is disabled and says why.
6
Every scenario this version handles. *Attachments:* an executable or a text file (refused by type), an empty file, a corrupt image, more than three images, a photo larger than the limit (shrunk), a transparent PNG (white background), remove before sending, the same file chosen twice, a drop with no files. *Sending:* the route refusing with 413 or 415. *Voice:* a clear question, silence, a blocked microphone, no support for the API, a double press. *And the stand-in model cannot see the image*: its reply only says an image arrived.
7
What Mitra can do now that it could not before: accept screenshots and pictures through a picker, a drop or a paste; shrink and check them; show them in the conversation; take a dictated question and let the person fix it. What is still missing: reading answers aloud (Topic 11.5 gives the sentence speaker; wiring it to a visible Listen control is Practice 3 below); keeping attachments across a page reload (Phase 4's storage would have to hold links, not data URLs); real image understanding (needs a real vision model on the route); Mitra answering simple things without a server (Phase 12, built in Topic 12.8); and the usage meter and daily limits that stop a user from sending a hundred images (Topics 13.1 and 13.2).

Explain it without notes

Walk through what happens between choosing a 12-megabyte screenshot and the model's answer.

Why do the words from dictation go into the text box?

Which checks live in the composer and which in the route, and why both?

Practice

Add a "Remove all" button to the chips list that revokes every URL and clears the list.

Make the route reject a request that has neither text nor a file with a 400 and a sentence, and show it in #error when the visitor sends an empty message.

Wire the Topic 11.5 speaker to Mitra: add a "Listen" button on each assistant message that speaks that message, and a Stop button that cancels.

Trade-offs

↔
Shrinking in the browser keeps requests small and cheap but adds code and CPU work on the user's device. Uploading the original to storage and resizing on the server gives you control and one place to fix things, and needs storage.
↔
Dictation with the browser's recogniser is free and instant to start; server transcription is the same everywhere. Mitra uses the browser's for the first version and keeps the text box as the single place where words are confirmed.

Done when you can

I can build a composer that takes images by picker, drop and paste.
I can validate, shrink and preview an image before sending it.
I can send text and file parts with sendMessage.
I can put dictated words in the box instead of sending them.
I can turn route refusals into sentences on screen.
I can say what Mitra still lacks and which phase adds it.

Command Palette

Words you'll meet

Step by step

01Start from Mitra

02Step 1: prepare an attachment

03Step 2: the Composer

04Step 3: send

05Step 4: dictate

06Watch it run

Try it yourself

Turn voice off

Lower the image limit

Let one more image in

Block the microphone

Code & diagrams

Break it on purpose

Send the dictation automatically

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can