PHASE 12Advanced ~28 min· topic 7 of 8

Topic 12.7

Chat Models in the Browser

In one line

Two routes put a chat model on the device: libraries such as WebLLM that you ship and run yourself, and the browser's own built-in AI APIs. Both need a large one-time download and capable hardware. The safe design is hybrid: try the device first and fall back to the server.

Think of it like this

A town with a doctor in the village and a hospital in the city. For a cut finger, you go to the village doctor: close, quick, no appointment. For something serious, or when the doctor is away, you go to the city. The village doctor must also know when to say "this is beyond me". A browser chat model is the village doctor. The server model is the hospital. A hybrid app sends each person to the right one.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

WebLLM: A library that downloads a chat model and runs it in the browser using WebGPU. Package @mlc-ai/web-llm.
Built-in AI APIs: AI features the browser itself provides to web pages, such as Chrome's Prompt, Summarizer and Translator APIs.
Prompt API: Chrome's built-in API for talking to a language model: LanguageModel.create(), prompt() and promptStreaming().
Origin trial: A time-limited program where a site opts in to an experimental browser feature. It may change or end.
availability(): A call that tells you if the model is unavailable, downloadable, downloading or available.
User activation: A recent click or key press by the user. Some browser actions, like starting a model download, require it.
Hybrid: Using the device first and the server as fallback, in one feature.
Engine: WebLLM's object that holds a loaded model and answers chat requests.

Step by step

01WebLLM, the shape of the code (not live)

WebLLM needs WebGPU and a large download, so it cannot run in the sandbox. This is its documented shape. Note that the answer arrives in the OpenAI-style chunk format, the same one you parsed by hand in Phase 2.

webllm.js (real library, not live)whole filejs

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("a-model-id-from-prebuiltAppConfig.model_list", {
  initProgressCallback: (report) => showProgress(report), // download and load progress
});

const chunks = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Where is my order?" }],
  stream: true,
});
for await (const chunk of chunks) {
  append(chunk.choices[0]?.delta?.content ?? "");
}

02Chrome's Prompt API, the shape of the code (not live)

The Prompt API sits behind a global called LanguageModel. Check it exists, ask for availability with the options you will use, and create the session from a click handler, since the download needs user activation.

prompt-api.js (real browser API, not live)whole filejs

if (!("LanguageModel" in self)) return useServer();

const availability = await LanguageModel.availability();
if (availability === "unavailable") return useServer();

// call this inside a click handler: the download needs user activation
const session = await LanguageModel.create({
  monitor(m) {
    m.addEventListener("downloadprogress", (e) => showProgress(e.loaded)); // 0 to 1
  },
});

const stream = session.promptStreaming("Where is my order?");
for await (const part of stream) append(part);

03Test every availability state

The first live example builds a stand-in LanguageModel that returns each state in turn, including a made-up future one, "something-new", which must be treated as unavailable. It also sends fake downloadprogress events (25, 50, 100 percent) through the same monitor hook, so the progress UI path is tested without a download.

The pattern is dependency injection: the code that decides takes the API object as an argument. Then your tests can pass a fake that returns any state, which is the only practical way to test a feature that depends on a browser trial.

04The hybrid, five cases

The second example runs: a question the device answers; a question the device is not sure about; a device that does not support it; a device model that crashes; and a server that is rate-limited (429). The first goes to the device. The next three go to the server with a log of why. The last ends with a plain error message and no answer, since both paths failed.

The server here is a serve() handler using the AI SDK's streamText with a scripted model; in a real app it is your route and a real model. The on-device model is a keyword stand-in, so you can see the decision logic, not model quality.

The hybrid, five casesdiagram

Rendering diagram…

05Tell the truth in the interface

Label the source of each answer. Offer a way to ask again with the other option ("Ask our server instead"). Ask before sending private text to the fallback (Topic 12.1). And show the first-run cost: "Setting up on-device AI, 0 percent", with a Cancel, and a way to carry on using the server meanwhile, so nobody waits on a download they did not ask for.

Try it yourself

1
Add a consent step
In the second example add a privateText flag to the call and, when the device failed and privateText is true, return { source: "none", text: "Ask the user before sending this to our server." } instead of calling the server.
Run a case with privateText: true and supported: false. This is the "ask first" branch of Topic 12.1, now in code.
2
A new state
In the first example, in the for loop at the bottom, replace "something-new" with "paused" and run it. describe() has no case for "paused".
It prints "paused -> Unknown state: treat it as unavailable." That default branch is what saves you when a browser adds a state your code has never seen.

Code & diagrams

Detect the built-in API and test every availability state

The page shows the real answers for this browser: whether `LanguageModel` exists and what `availability()` says. They differ between browsers and machines, so they are never printed to the console. The console uses a stand-in object with the same method names, so every state can be tested without a multi-gigabyte download.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

the check itself never throws: true
unavailable -> Hide the feature or use the server. This device cannot run it.
downloadable -> Offer a button. The model must be downloaded first (needs a click).
downloading -> Show a progress bar. A download is already running.
available -> Use it now. No progress UI needed.
something-new -> Unknown state: treat it as unavailable.
download 25%
download 50%
download 100%
stand-in reply to: hello

On-device first, server as the fallback

The on-device model is a keyword stand-in. The server uses the AI SDK with a scripted fake model that always gives the same reply, whatever the question, so the point is the decision logic.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

Can I get a refund?
  device: You can return any unopened item within 7 days of delivery.
What do you sell?
  device not sure, asking the server
  server: The server model says: we sell tea, snacks and gift boxes.
Can I get a refund?
  device failed (unsupported), asking the server
  server: The server model says: we sell tea, snacks and gift boxes.
Can I get a refund?
  device failed (model crashed), asking the server
  server: The server model says: we sell tea, snacks and gift boxes.
server busy, shipping?
  device failed (unsupported), asking the server
  none: Sorry, I cannot answer right now (server said 429).

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the device's first answer

In the second example change if (local.confidence >= 0.5) return to return so the device's answer is always used.

terminal

$ The second question's lines

── what you'll see ──

What do you sell?

device:

Myth vs fact

Myth

Built-in browser AI is available in every browser.

Fact

Chrome's documentation covers Chrome only, and the APIs there are at different stages: stable, developer trial or origin trial. Other browsers have their own plans or nothing. Feature-detect and keep the feature optional.

Myth

A chat model in the browser is as good as a server model.

Fact

A model small enough to download is far less capable than the large models on servers, and may be weak at long answers, facts and other languages.

Myth

If availability() says available, the call cannot fail.

Fact

It can still throw (out of memory, the browser stopping it, a policy). Always catch and fall back.

Myth

The download happens quietly in the background.

Fact

It is large and needs the user's click. Say what is happening and show progress, or the page looks broken.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Wrap both routes behind one interface, for example ask(messages, { signal }) returning a stream of text. The rest of the app does not care whether the words come from WebLLM, the Prompt API or your server, and you can add or remove a route without touching the UI. Cancel and partial answers (Phase 3) then work the same everywhere.
▸
Check the requirements honestly before offering. Chrome's own list (22 GB free, more than 4 GB of VRAM or 16 GB RAM with 4 cores, an unmetered connection) means many devices will say unavailable. Plan for most of your users to take the server path, and treat the device path as an upgrade for those who can use it.
▸
Measure both paths. Log which route answered and how often users retry or rephrase after each (Phase 13). The share of device answers and the satisfaction of each tells you whether the extra complexity pays for itself.
▸
Keep the server fallback tested and current. It is the path for your oldest devices and for every moment the device path breaks after a browser update. A fallback nobody exercises is a bug waiting for its first user.

Remember this

1
The problem this solves: Topics 12.1 to 12.6 used small models for narrow jobs. A chat model that can write sentences is much larger. This topic shows what exists today, how big the catch is, and the one design that works for every user: on-device first, server as the fallback, with the interface honest about which one answered.
2
Route 1: a library that runs a chat model for you. WebLLM (npm package @mlc-ai/web-llm, version 0.2.86 when this was written) runs language models in the browser, accelerated with WebGPU. You create an engine with CreateMLCEngine(model, { initProgressCallback }), which downloads and loads the model, then call engine.chat.completions.create({ messages, stream: true }), the same shape as the OpenAI-style chat request from Topic 1.1, and read the chunks with for await. Its README says the models you can use are a subset of a published list, available as prebuiltAppConfig.model_list; it can run inside a Web Worker (CreateWebWorkerMLCEngine on the page and WebWorkerMLCEngineHandler in the worker) and it supports four cache backends set by AppConfig.cacheBackend: "cache" (default), "indexeddb", "opfs" and "cross-origin". Choose the model from that list and its model card, and check the size before you promise anything.
3
Route 2: the browser's built-in AI. Chrome ships AI models and exposes them as JavaScript APIs, so your site does not download the model: the browser does, and shares it between sites. As read on developer.chrome.com (October 2026): the Translator, Language Detector and Summarizer APIs are available from Chrome 138 stable; the Writer, Rewriter and Proofreader APIs are marked "developer trial" in that table (the page's own text calls the Proofreader an origin trial, so the page is not even consistent with itself); and the Prompt API (LanguageModel) is listed for web pages from Chrome 148, and for extensions from Chrome 138, with an origin trial for its sampling parameters. This page changes often, so read the current status table before you build on any of them. Only Chrome is documented there; other browsers have their own plans or nothing.
4
The Prompt API's catches, from Chrome's own documentation. Hardware: at least 22 GB of free space on the volume with the Chrome profile, and either a GPU with strictly more than 4 GB of VRAM or 16 GB of RAM and 4 or more CPU cores; an unlimited or unmetered connection for the first download; Windows 10 or 11, macOS 13 or newer, Linux, or ChromeOS on Chromebook Plus. Downloading and creating a session need a user gesture (a click). It is available in top-level windows and same-origin iframes by default (a cross-origin iframe needs the attribute allow="language-model"), and not in Web Workers yet. If free space later drops below 10 GB, Chrome removes the model and downloads it again when the requirements are met, so availability() can change during a user's day. These numbers are Chrome's published requirements today, not a promise for the future.
5
Four availability states. await LanguageModel.availability() returns "unavailable" (this device cannot run it: hide the feature or use the server), "downloadable" (offer a button; the model must be downloaded), "downloading" (show progress) or "available" (use it now). LanguageModel.create({ monitor(m) { m.addEventListener("downloadprogress", (e) => ...) } }) reports e.loaded (a fraction from 0 to 1). Pass availability() the same options you pass to prompt() or promptStreaming(). The first live example checks whether the real browser has LanguageModel and what its availability() says, inside try … catch, and shows the answers on the page (they differ between browsers and machines), then tests all four states with a stand-in object of the same shape. MDN marks the API "limited availability".
6
Why "experimental" matters. An API in a trial may change shape or disappear, may exist in only one browser, and may be switched off by the user's settings or company policy. Wrap every use in feature detection, keep the whole feature optional, and never make the only path to a core task depend on it.
7
The hybrid pattern. Order: (1) Is on-device support present and available? (2) Try it. (3) If it throws, or its confidence is low, or the job needs fresh facts, ask the server. (4) If the server also fails, show a clear error. And tell the user which one answered ("on this device", "from our server"), since the privacy and accuracy differ. The second live example runs five cases through exactly this order.

Explain it without notes

Name two ways to run a chat model on the device, and the main limit of each.

What are the four availability() states, and what should the interface do in each?

Describe the hybrid pattern and when each step falls back.

Why wrap the on-device model and the server model behind one interface?

Practice

Write ask(question, routes) where routes is an array of async functions tried in order; it returns the first answer with the index of the route, and an error only if all fail.

Add a timeout of 300 ms to the on-device call in the second example so a stuck device model falls back to the server.

Trade-offs

↔
Built-in browser AI saves you the download and the hosting, but ties the feature to one browser's trials and rules. A library like WebLLM works across WebGPU browsers but you carry the download, caching and updates.
↔
Hybrid gives the best coverage and doubles the paths to build, test and monitor. Device-only is simpler and excludes many users. Server-only is simplest and gives up the on-device benefits.

Done when you can

I can describe WebLLM's flow and the browser's built-in AI APIs and their current status.
I can list the requirements and limits Chrome documents for the Prompt API.
I can handle all four availability states plus an unknown one.
I can write the hybrid order: device, then server, then a clear error.
I can say what the interface tells the user about which path answered.

Command Palette

Words you'll meet

Step by step

01WebLLM, the shape of the code (not live)

02Chrome's Prompt API, the shape of the code (not live)

03Test every availability state

04The hybrid, five cases

05Tell the truth in the interface

Try it yourself

Add a consent step

A new state

Code & diagrams

Break it on purpose

Trust the device's first answer

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can