Command Palette

Search for a command to run...

Hectal
Phase 11Intermediate12 of 16 in AI Frontend

Files, Images and Voice

Let people attach files and screenshots, speak their question, and hear the answer: with limits, previews, permissions and every failure handled on screen.

So far Mitra only reads and writes text. People do not only type: they want to show a screenshot of a broken order page, attach a receipt, say a question out loud while their hands are busy, or hear the answer read back. This phase adds those inputs and outputs to the browser side of an AI feature.

Each of them brings its own limits and its own ways to fail. A file can be too big, empty, the wrong type, or not what its name says. A photo can be twelve megabytes. A microphone can be blocked, missing, or already in use. A voice can be silent until the user taps. You will check each one before it costs money, show its state on screen, and say in plain words what went wrong.

The sandboxed preview has no camera, no microphone and no real file picker, so the examples create files in code, draw test images on a canvas, and show the blocked and unsupported paths live. The working microphone and speaker paths are shown as code and checked against MDN. The stand-in model cannot see images or hear audio: it is scripted, and each example says so.

0/6 · 0%
6 topics ~4 h 24 code blocks & diagrams
Start with the first topic
1
11.1

File Attachments

A file reaches your page through a picker, a drop or a paste. Check its size, type and contents before you accept it, show a preview, send it as a file part of the message, and handle an upload that fails without losing the other files.

37 min 4 code practice

2
11.2

Images as Input

A model reads an image you send in the message, as a data URL inside the request or as a link to an uploaded file. Shrink the photo in the browser first: it cuts upload time, request size and cost, and you decide which formats are allowed before the provider refuses one.

35 min 4 code practice

3
11.3

Image Generation UX

Making a picture takes many seconds and fails in more ways than text does. Reserve the space so the page does not jump, show honest progress, give a refusal a different screen from an error, keep the pictures that did succeed, and describe every generated image with alt text.

39 min 4 code practice

4
11.4

Speech Input

Speaking a question needs the user's permission for the microphone, a recorder that starts and stops cleanly, and a way to turn sound into text. Each step can be refused, missing or empty, and the screen must say which one happened and what to do next.

41 min 5 code practice

5
11.5

Speech Output

To read an answer aloud while it is still arriving, cut the stream into whole sentences, clean the markup out, and queue each sentence for a voice. The page must stop the voice at once when the user asks, ignore the noise of its own cancelling, and cope with browsers that will not play sound until the user taps.

39 min 5 code practice

6
11.6

Build: Mitra Sees and Listens

Mitra gets two new inputs. A customer can attach a screenshot of a problem (checked, shrunk and previewed before it is sent) and can dictate a question with their voice (the words go into the box for correction). Every refusal has its own message.

41 min 2 code practice

Practise this phase

Hands-on exercises: write the code, see your page live, and press Check to have it tested automatically.

1

Check a File Before You Accept It

A file picker lists the files a person chooses. Write validate(file) so it returns a sentence when the file must be refused, and null when it is fine. The page already shows the accepted names in #list and the sentences in #error.

Beginner
2

Work Out the Shrunk Size

Before sending a photo, Mitra shrinks it so its longest side is at most max pixels. Write fitWithin(w, h, max) that returns { w, h }. Keep the shape (scale both sides by the same factor), round to whole pixels, never enlarge a small image, and never return less than 1 pixel.

Beginner
3

Say Why the Microphone Failed

The page tries to open a stand-in microphone in five situations chosen with the #case menu. Write describe(outcome) which returns the sentence for #say. outcome is "ok", "unsupported", or the name of the error the browser would raise.

Intermediate
4

Cut an Answer into Spoken Sentences

Write speakable(text, min) that returns an array of sentences to speak. Split with Intl.Segmenter (locale "en", granularity: "sentence"), trim each piece, and join short pieces with the next one until the text is at least min characters, so Dr. is not spoken alone. Whatever is left at the end is the last item, even if it is short.

Intermediate
5

Upload a File and Say What Happened

The route /api/upload is already running in the page: it answers 413 for a file bigger than 1,000 bytes and fails once (500) for a file named flaky.png. Write the click handler for #send.

Intermediate