PHASE 12Advanced ~23 min· topic 1 of 8

Topic 12.1

Why Run a Model in the Browser

In one line

Running a model on the user's device keeps their text private, costs you nothing per request and works offline, but the model is small, has to be downloaded and does not run everywhere. The skill is deciding, per job, whether it belongs on the device or on the server.

Think of it like this

A pocket calculator and a bank's computer. For adding up a bill, the pocket calculator is perfect: it is in your hand, it works in a tunnel, and nobody else sees your numbers. For checking the exchange rate right now, you need the bank's computer. A model in the browser is the pocket calculator: fast, private and limited. A model on the server is the bank's computer: powerful and up to date, but far away. Good products use both.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

On-device: Running the model on the user's own computer or phone, inside the browser, instead of on your server.
Inference: Using a trained model to produce an answer. Training is how the model was made; inference is every use of it afterwards.
Small model: A model made small enough to download and run on an ordinary device. Smaller means less capable.
WebAssembly (WASM): A compact code format that browsers run quickly. Most on-device models run on it by default.
WebGPU: A browser feature that lets web pages use the graphics chip for general computing, such as running a model.
Confidence: A number a classifier gives for how sure it is. Low confidence is the signal to ask the server.
Fallback: What the app does when its first choice cannot work, such as asking the server when the device cannot run the model.
Stand-in: A tiny fake used in an example in place of the real thing. The stand-in model here counts words; it is not a neural network.

Step by step

01Name the reason first

Before any code, write down which of the four reasons applies: privacy, cost, offline or latency. If none does, the server is simpler. A team that moves a job to the browser "because it is cool" ends up with a download, a device matrix and a fallback, and no benefit.

02Turn the decision into a function

The first live example is one plain function, whereToRun(job, device), with the rules in order. Fresh facts always go to the server. Private text stays on the device if the device can run the model, and otherwise the app must ask the user before sending it away. Long answers go to the server.

Seven jobs print seven decisions. Because it is a function, you can test it, and the product team can read and argue with the rules.

03Meet a small model's limits

The second example is a stand-in classifier: it counts known words and reports a score. It gets "where is my order" right. It also misses the clear question "can I return this" because only one word in four is known, and it cannot read Hindi or emoji at all.

A real small model reads meaning better than a word counter, but it has the same shape: an answer plus a confidence, and a range where it is out of its depth. The score threshold is how you notice that.

04What to do with an unsure answer

When confidence is below the line, return "unknown" and let the app choose: ask the server, ask the user, or show the plain search. The breakIt below removes the threshold and the stand-in cheerfully labels a poem request as an order question. Wrong and confident is the worst combination, and a threshold is the cheapest guard.

What to do with an unsure answerdiagram

Rendering diagram…

05What it looks like in production

Teams that ship this track two numbers: how often the device answer is used, and how often users then rephrase or override it. If the device answers 80 percent of questions and nobody complains, the server bill dropped. If users keep rephrasing, the threshold is too low. Phase 13 covers measuring AI features.

Try it yourself

1
Add a rule
In the first example add a rule: if job.regulated is true, the answer is "server: needs an audit log". Add a job with regulated: true. Where in the function must the rule go so it wins over "device: cheap and instant"?
Order matters: rules that must always win go first. A rule at the bottom is never reached for jobs the earlier rules already caught.
2
Lower the bar
In the second example change 0.3 to 0.2 and run it. "can I return this" now gets a label.
Then add the inputs "return return return" and "tea". They give the same answer at both bars (score 1 and score 0). Only an input that scores between the two bars, like 0.25, changes sides. Choosing a threshold is choosing which in-between inputs you let through, so it is choosing which kind of mistake you prefer.

Code & diagrams

Where should this job run?

Plain logic, no model. The point is that the decision is a function you can test, with the fallback rules written down in order.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

tag a support message -> device: cheap and instant
where is my order ORD-1042 -> server: the answer depends on data the device does not have
summarise my private notes -> device: the text never leaves the browser
summarise my private notes (old phone) -> ask first: private text, but this device cannot run the model
write a 600 word product story -> server: small models write long answers badly
classify intent, user is on a train -> device: there is no network to use
classify intent (old phone) -> server: this device cannot run the model

A small stand-in model: what it gets right and where it is out of its depth

This is a stand-in, not a neural network: it counts words from a short list. A real small model is better at meaning, but it also has a confidence and a range where it is wrong. Note the Hindi and emoji lines: the stand-in finds no Latin words and says so.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"where is my order" -> {"label":"order_status","score":0.5}
"can I return this" -> {"label":"unknown","score":0.25,"why":"too few known words"}
"what time do you open" -> {"label":"hours","score":0.4}
"" -> {"label":"unknown","score":0,"why":"no Latin words found"}
"👍" -> {"label":"unknown","score":0,"why":"no Latin words found"}
"मेरा ऑर्डर कहाँ है" -> {"label":"unknown","score":0,"why":"no Latin words found"}
"compare your Darjeeling to Assam and write me a poem about it" -> {"label":"unknown","score":0,"why":"too few known words"}
"ORD-1042 ORD-1042 ORD-1042" -> {"label":"order_status","score":0.5}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the confidence threshold

In the second example, delete the line if (score < 0.3) return { label: "unknown", ... }; and run it.

terminal

$ The line for the poem request

── what you'll see ──

"compare your Darjeeling to Assam and write me a poem about it" -> {"label":"order_status","score":0}

Myth vs fact

Myth

On-device AI is always faster than a server.

Fact

Only after the model is loaded. The first use pays a download and a start-up. A small, quick task can win; a heavy one on a weak phone loses to a server.

Myth

If it runs in the browser, it is automatically private.

Fact

Only if you do not also send the text or the result somewhere else, such as logs or analytics. The model being local does not stop the rest of your app from sending data out.

Myth

A model that runs on the device can replace the server model.

Fact

Small models handle narrow jobs. For long writing, reasoning and fresh facts you still need a server model, so most real products are hybrid.

Myth

Every device can run it.

Fact

Devices differ in memory, browser version and graphics support. The app must detect and fall back (Topic 12.3).

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Privacy claims need care. "Your text never leaves your device" is a strong promise: it has to hold for every path, including the fallback. If the fallback sends text to the server, the interface must say so before it happens, or ask for consent, as the "ask first" line of the first example does.
▸
A model is a dependency with a version. If you change it, answers change: the same input may get a different label. Keep the model version in your logs and in your cache key (Topic 12.4), and test the fallback path regularly because it runs for the users on the oldest devices.
▸
Do not decide by the developer laptop. The people who most need the fallback are on old phones, in battery saver mode, or on a metered connection where a large download is a cost to them. Test on the weakest device you support.

Remember this

1
The problem this solves: Every question you send to a server model travels over the network, costs money, and shows your user's text to another company. For many small jobs (sorting a message into a category, finding the right help article, checking a form) that is more than the job needs. This topic gives you the reasons to run a model on the device, the honest limits of small models, and a way to decide where each job belongs.
2
What "on the device" means. The model's files are downloaded to the browser and the computation runs there: on the CPU through WebAssembly (a compact code format that browsers run at near-native speed) or on the graphics chip through WebGPU (Topic 12.3). Your server is not involved in the answer at all. Nothing is sent anywhere, so there is no request to fail, rate-limit or bill.
3
Four reasons to do it. *Privacy:* the text stays in the browser, which matters for notes, health or money. *Cost:* the user's device does the work, so your bill does not grow with use. *Offline:* once the model is saved on the device, it works with no network. *Latency:* no round trip to a server, so a small task can answer in a moment. Each reason is real only for some products; name the one you are buying before you pay the price.
4
The price. The model is a download, usually many megabytes, that the user waits for once (Topic 12.4). Devices differ: an old phone may not run it at all (Topic 12.3). Inference uses the user's battery and can slow the page if done on the main thread (Topic 12.5). And you now ship, version and cache model files the way you ship code.
5
What small models can and cannot do. A model small enough to download is far smaller than the large models on servers. Small models are good at narrow, well-defined jobs: classifying text into a few labels, finding which help article is closest to a question (Topic 12.6), detecting a language, or short, simple rewrites. They are weak at long, careful writing, deep reasoning, many facts and unusual languages. They also do not know anything that happened after they were made, and they do not know your private data unless you give it to them.
6
A good rule. Use the device for jobs that are small, private or frequent, where a wrong answer is cheap and a fallback exists. Use the server for jobs that need fresh facts, long answers or high accuracy. The live examples make this a function you can test, then show how a small stand-in model gets out of its depth, so that you build the fallback from the start.

Explain it without notes

Give four reasons to run a model in the browser, and one reason not to.

What kinds of jobs suit a small on-device model, and what kinds do not?

Why should an on-device classifier return a confidence, and what do you do with a low one?

A product says "your notes never leave your device", but its fallback calls a server model. What is wrong and how do you fix it?

Practice

Extend whereToRun so a user setting device.metered (a metered connection) sends large downloads away: if the model is not yet downloaded and the connection is metered, the answer is "server: avoid a large download on a metered connection". Where do you put the rule?

Write a function labelWithFallback(text) for the stand-in classifier that returns the label when the score is at least 0.3 and the string "ask server" otherwise, and log it for three inputs.

Trade-offs

↔
Privacy and cost improve on the device, but you accept a download, a device matrix and a weaker model. The server is simpler and stronger but sees the data and bills for it. Most products end up hybrid, with the extra code that implies.
↔
A higher confidence threshold sends more to the server (safer, costlier). A lower one answers more on the device (cheaper, more mistakes). Pick it with real examples, not by feel.

Done when you can

I can name four reasons to run a model on the device and the costs it brings.
I can say which jobs suit a small model and which need a server model.
I can write the placement decision as a testable function with a clear rule order.
I can explain why a classifier needs a confidence threshold and an "unknown" path.
I can say what the user must be told before any private text goes to a fallback.

Command Palette

Words you'll meet

Step by step

01Name the reason first

02Turn the decision into a function

03Meet a small model's limits

04What to do with an unsure answer

05What it looks like in production

Try it yourself

Add a rule

Lower the bar

Code & diagrams

Break it on purpose

Remove the confidence threshold

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can