PHASE 11Intermediate ~35 min· topic 2 of 6

Topic 11.2

Images as Input

In one line

A model reads an image you send in the message, as a data URL inside the request or as a link to an uploaded file. Shrink the photo in the browser first: it cuts upload time, request size and cost, and you decide which formats are allowed before the provider refuses one.

Think of it like this

Sending a photo of a document to a friend by message. A 12-megabyte original takes a long time on a weak connection, and your friend only needs to read the text. So you send a smaller copy. Shrinking an image before sending it is the same favour to your user, your server and your bill.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Data URL: An address that holds the data inside it: data:image/jpeg;base64, followed by the bytes written as text.
Base64: A way to write bytes as plain text. Every 3 bytes become 4 characters, so the text is a third bigger than the data.
Object URL: A short temporary address for a file in memory, from URL.createObjectURL. It only works in this page and tab.
Canvas: A drawing surface in the page. Drawing an image onto a smaller canvas resizes it.
ImageBitmap: A decoded image ready to draw, made by createImageBitmap. Call close() when finished.
EXIF orientation: A note inside a photo that says which way is up, because the camera stores the pixels as the sensor saw them.
Lossy format: A format that throws away detail to be smaller, such as JPEG and WebP. PNG is lossless.
Pixel size: Width and height in pixels. Many providers use it, together with the file size, when they limit or price an image: check yours.

Step by step

01See what a data URL costs

The first example turns a 3,000-byte file into a data URL with FileReader.readAsDataURL. The base64 part is exactly 4 * ceil(3000 / 3) characters, so 4,000 for 3,000 bytes. An object URL for the same file is a short blob: address, but it is only valid in this page, and after URL.revokeObjectURL it cannot be fetched.

The same example puts each form in a request body and prints the length. The empty file is the edge case: its data URL is only the header, data:image/jpeg;base64,, which looks like an image and is not.

02Compute the new size

The second example starts with fitWithin(width, height, max). It returns the same size when the image already fits (no enlarging), scales both sides by the same factor otherwise, rounds, and never returns less than one pixel. It prints six cases, including a very wide strip that would round to zero without the Math.max(1, …).

fit-within.jswhole filejs

function fitWithin(w, h, max) {
  const longest = Math.max(w, h);
  if (longest <= max) return { w, h, scaled: false };
  const k = max / longest;
  return { w: Math.max(1, Math.round(w * k)), h: Math.max(1, Math.round(h * k)), scaled: true };
}

03Draw it smaller and export

shrink() decodes the file, computes the size, paints a white background, draws the bitmap on the smaller canvas and exports JPEG. The example shrinks a 3000 by 2000 picture to 1024 by 683 and prints that the new file is smaller. Then it shrinks a transparent PNG and prints the top-left pixel, which is white because of the fill. Remove the fill (Break it) and it is black.

Draw it smaller and exportdiagram

Rendering diagram…

04Refuse what cannot work

The third example opens six files: a good PNG, a cut-off PNG, an empty one, an SVG, a file with no type, and one with the type image/heic. Each gets a reason and a sentence for the user. The decode test catches the damaged ones that the type check lets through.

05When the provider says no anyway

The fourth example sends an image part to a route that behaves like a provider that does not accept image/heic. The provider answers 400. The route passes a readable message to the browser and useChat moves to error. A browser check is a kindness; the provider's answer is the truth, and your UI must handle it.

Try it yourself

1
Change the limit
In the second example change 1024 to 512 in the call shrink(original, 1024) and run it.
The sent size becomes 512x341. Half the width and height is a quarter of the pixels, so the model reads much less. Test with a real screenshot of small print to find the smallest size that is still readable.
2
Quality number
In shrink, change 0.85 to 0.3, then compare the blob sizes by logging small.blob.size for both.
Lower quality gives a smaller file. In a real photo, you will see blocks and blur around text first. Quality has no effect if you export image/png.
3
Allow GIF
In the third example add "image/gif" to ALLOWED and add a file of that type with a few bytes.
It now passes the type check and fails the decode test with cannot-decode. A type allow-list says what you accept; the decode test says whether this particular file works.

Code & diagrams

Data URL or object URL

The same bytes in two forms. A data URL travels inside every request that includes the message. An object URL is tiny but only lives in this page.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

file bytes: 3000
data URL starts: data:image/jpeg;base64,
base64 characters: 4000, 4 * ceil(3000 / 3) = 4000
object URL starts: blob:, short: true
after revoke, fetching it fails: TypeError
request body with the data URL: 4084 characters
request body with a link: 96 characters
empty file as a data URL: data:image/jpeg;base64,

Shrink a photo before sending

Six size cases first, then a real 3000 by 2000 picture drawn on a canvas and shrunk. Byte sizes depend on the browser's encoder, so the example prints comparisons, not numbers.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

3000x2000 -> 1024x683
400x300 -> 400x300 (kept, never enlarged)
1x1 -> 1x1 (kept, never enlarged)
2000x4000 -> 512x1024
5000x10 -> 1024x2
100000x10 -> 1024x1
sent: 1024x683 image/jpeg
smaller than the original: true
transparent PNG, top-left pixel after shrink: 255,255,255,255
zero-size canvas exports: null

Refuse an image that cannot work

The type check catches formats you do not allow. The decode test catches files that look right and are not. SVG is refused on purpose: it is text that can carry scripts and is not a picture the model reads.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

receipt.png -> ok 20x20
cut-off.png -> cannot-decode (InvalidStateError)
  tell the user: This image looks damaged or cut off. Try another copy.
empty.png -> cannot-decode (InvalidStateError)
  tell the user: This image looks damaged or cut off. Try another copy.
logo.svg -> format-not-supported
  tell the user: This image format is not supported. Save it as JPEG or PNG and try again.
mystery -> unknown-type
  tell the user: We could not tell what kind of file this is. Save it as JPEG or PNG and try again.
IMG_0042.heic -> format-not-supported
  tell the user: This image format is not supported. Save it as JPEG or PNG and try again.

The provider says no to an image

The route stands in for a provider that refuses HEIC. The scripted visitor sends the second image when the status first becomes error, so the order never depends on timing.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

status: ready
status: submitted
status: error | error: Unsupported image format: image/heic
status: submitted
status: streaming
status: ready

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget the white background

In the second example, delete the two lines that set g.fillStyle = "#fff"; and call g.fillRect(0, 0, size.w, size.h); inside shrink. Run it.

terminal

$ Console, the transparent PNG line

── what you'll see ──

transparent PNG, top-left pixel after shrink: 0,0,0,255

Myth vs fact

Myth

A bigger image always gives a better answer.

Fact

Past a certain size the model gains nothing, and you pay for it. Choose the smallest size at which the text you care about is still readable.

Myth

A data URL is the same size as the file.

Fact

Base64 writes every 3 bytes as 4 characters, so it is about a third bigger, and the chat resends it with every turn.

Myth

canvas.toBlob(cb, "image/webp") always gives a WebP.

Fact

If the browser cannot encode the type you asked for, it quietly exports PNG instead. Check blob.type.

Myth

If the file has an image type, the model can read it.

Fact

The file can be damaged, and your provider accepts only some formats. Decode it in the browser and expect a 400 from the provider anyway.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Resizing is CPU work on the main thread. A 12-megapixel photo can freeze the page for a moment on a slow phone. For several large files, process them one at a time, show progress, or move the work to a Web Worker with OffscreenCanvas (Topic 12.5 covers workers).
▸
Providers also limit how many images one message may carry and how large each may be, and those limits change. Keep your own count (MAX_FILES) and edge length (MAX_EDGE) as named constants, set them at or below what your provider documents today, and let the route enforce them (Topic 11.6).
▸
Free memory. Call bitmap.close() after drawing, and revoke object URLs. A page that handles ten 12-megapixel photos without releasing them can be killed by the browser on a phone.
▸
Keep a copy-free path for small images: if the file is already under your size and pixel limits and is an allowed type, send it as it is. Re-encoding a small JPEG only makes it blurrier.
▸
Alt text for the user's own image: show the file name and a short caption, not the model's guess. The image in the conversation should still make sense to someone using a screen reader, and a model's description is not always right (Topic 11.3 covers alt text).

Remember this

1
The problem this solves: A phone photo is often 4000 pixels wide and several megabytes. Sent as it is, the upload is slow, the request may hit a size limit, the model reads far more detail than it needs, and many providers charge more input for bigger images. Some formats are not accepted at all. This topic shows how an image travels, how to shrink it in the browser, and how to refuse the ones that cannot work.
2
Two ways to send an image. (1) Inside the message as a data URL: the file's bytes written as text, data:image/jpeg;base64,/9j/4AAQ…. Nothing else is needed, but the text is about one third bigger than the file (base64 turns every 3 bytes into 4 characters), and it travels in every request that includes the message. (2) As a link: you upload the file once to storage and send its address in the file part's url. The request stays small, and the link can be reused, but you need storage, a route and a clean-up plan, and the provider must be able to download it (a link only your logged-in user can open will not work).
3
Why the conversation makes it worse. A chat resends earlier messages on every turn (Topic 4.3). An image attached in turn 1 as a data URL is sent again in turn 2, 3 and 4. A five-turn chat with one 3-megabyte image sends that image five times. Shrinking, or sending a link, matters more the longer a conversation runs.
4
Shrink with a canvas. Decode the file with createImageBitmap(file), which gives the pixel size and handles photo rotation stored in the file (the default option imageOrientation: "from-image" applies the EXIF orientation). Compute a new size that keeps the shape: scale so the longest side is at most your limit, never enlarge, never go below one pixel. Draw the bitmap onto a smaller <canvas> and export with canvas.toBlob(callback, "image/jpeg", 0.85). Exporting also drops hidden metadata such as GPS position.
5
Mind the format you export. toBlob falls back to PNG if the browser does not support the type you asked for, so check blob.type on the result. JPEG has no transparency: a transparent PNG exported as JPEG gets a black background (in the Chromium used here). Fill the canvas with white first. PNG keeps sharp text and flat colours well and is larger for photos. JPEG is the usual choice for photographs and screenshots with a lot of detail. The quality number from 0 to 1 only applies to lossy formats.
6
Not every file named like an image is one. A file can be cut off, empty or damaged, and createImageBitmap then rejects (here with InvalidStateError). Formats your provider does not accept (a typical short list is PNG, JPEG, WebP and GIF; read the current one in your provider's documentation) get an error from the provider, often a 400. Decide your allow-list yourself, try to decode in the browser, and say in plain words what to do: "Save the image as JPEG or PNG and try again".
7
Cost. A provider turns an image into input tokens by its own rule and charges for them like text. Many base this on the image's size, but the rule differs and changes, so check your provider's documentation before you rely on it. A smaller image usually costs less and always uploads faster, but too small and the model cannot read small print in a screenshot. Pick the limit by testing real screenshots, and put it in one constant. Look up the current numbers instead of remembering any.

Explain it without notes

How can an image be sent to a model, and what does each way cost?

Why is a data URL bigger than the file, and why does it matter more in a long chat?

How do you resize an image in the browser, and what are the edge cases?

Why can an image that passes your type check still fail?

Practice

Write bytesToMB(n) and use it to log how many megabytes the second example saved, as a percentage, without printing byte counts.

Add a maxMegapixels rule to openImage: refuse an image with more than 40 million pixels with a helpful message, before any resizing.

Make shrink keep PNG when the input is a PNG with transparency, and JPEG otherwise.

Trade-offs

↔
Data URLs make the whole feature work with one route and no storage. Uploads plus links scale better and keep requests small, but add storage, expiry, access control and clean-up. Start with data URLs and shrunken images; move to links when requests or conversations get large.
↔
Aggressive shrinking saves money and time and can make small print unreadable. Treat the limit as a product decision, tested with the images your users really send.

Done when you can

I can explain why a data URL is a third bigger than its file.
I can resize an image with a canvas and keep its shape.
I can avoid a black background when exporting JPEG.
I can refuse a damaged or unsupported image with a clear message.
I can handle a provider's 400 for an image.
I can say what shrinking saves in upload time, request size and cost.

Command Palette

Words you'll meet

Step by step

01See what a data URL costs

02Compute the new size

03Draw it smaller and export

04Refuse what cannot work

05When the provider says no anyway

Try it yourself

Change the limit

Quality number

Allow GIF

Code & diagrams

Break it on purpose

Forget the white background

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can