Command Palette

Search for a command to run...

Hectal
Phase 12Advanced13 of 16 in AI Frontend

AI That Runs in the Browser

Run small models on the user's own device: Transformers.js, WebGPU and WebAssembly, downloads and caches, workers, on-device search, and a safe fallback to the server.

Until now every model lived on a server. This phase moves some of the work into the browser, onto the user's own device. That can mean privacy (the text never leaves the page), no per-request cost, answers that still work offline, and no wait for the network. It also means new problems: the model is a download of many megabytes, not every device can run it, and heavy computing can freeze the page.

You will learn the tools (Transformers.js and ONNX Runtime Web, WebGPU and WebAssembly, WebLLM, and the built-in browser AI of Chrome) and the engineering around them: detecting what a device can do, downloading and caching a model with honest progress, keeping the page responsive with Web Workers, and searching by meaning on the device. The most important idea of the phase is the hybrid: try the device first, and fall back to the server when the device cannot, fails, or is not sure.

The sandbox in this site's examples is not set up to download a model, so the live examples teach every mechanism with small real code and a clearly labelled stand-in for the model (a word counter or a keyword check, not a neural network). The sandbox itself is a real browser frame, though, so the live examples also run real WebAssembly, real Web Workers and real feature checks. The real libraries appear in code blocks that do not run here, with names and versions checked against their documentation. For a real model running live in your browser, the RAG course (/rag) has one.

0/8 · 0%
8 topics ~3 h 17 code blocks & diagrams
Start with the first topic
1
12.1

Why Run a Model in the Browser

Running a model on the user's device keeps their text private, costs you nothing per request and works offline, but the model is small, has to be downloaded and does not run everywhere. The skill is deciding, per job, whether it belongs on the device or on the server.

23 min 2 code practice

2
12.2

Transformers.js and ONNX Runtime Web

Transformers.js lets you run a trained model in the browser with one call, pipeline(task, model), and runs it on ONNX Runtime Web. You choose a task, a model and options, you wait for the files to load, and then you call the result like a function.

26 min 3 code practice

3
12.3

WebGPU and WebAssembly Back Ends

A model in the browser runs on one of two back ends: WebAssembly on the CPU, which almost every browser has, or WebGPU on the graphics chip, which is faster but not everywhere. You detect what the device offers, pick one, and keep a fallback down to "use the server".

25 min 2 code practice

4
12.4

Downloading and Caching Models

A browser model is a large download. Show honest progress, save the files in the Cache API so the next visit is instant, handle a full disk and an interrupted download, and put a version in the cache name so old files can be replaced.

28 min 2 code practice

5
12.5

Keeping the Page Responsive

Running a model on the page's main thread freezes the page: no clicks, no typing, no animation. A Web Worker runs the model on another thread and talks to the page by messages. You learn messages, request ids, progress, transferables, cancelling and errors.

27 min 2 code practice

6
12.6

Embeddings and Semantic Search On-Device

An embedding turns a piece of text into a list of numbers so that texts with similar meaning get similar numbers. With a small embedding model in the browser you can search help articles by meaning, as the user types, with no server and no text leaving the device.

25 min 2 code practice

7
12.7

Chat Models in the Browser

Two routes put a chat model on the device: libraries such as WebLLM that you ship and run yourself, and the browser's own built-in AI APIs. Both need a large one-time download and capable hardware. The safe design is hybrid: try the device first and fall back to the server.

28 min 2 code practice

8
12.8

Build: Mitra Works Offline for Simple Things

Mitra learns to answer the simple, common questions (returns, shipping, opening hours) on the device, with no network, and to send everything else to the server. The interface says where each answer came from and what is still possible offline.

27 min 2 code practice

Practise this phase

Hands-on exercises: write the code, see your page live, and press Check to have it tested automatically.

1

Pick the Back End

Five pretend devices are listed. Each button picks one and shows the back end your code chooses in #backend. Finish pickBackend(env) so it returns "webgpu", "wasm" or "server".

Intermediate
2

Show Download Progress

Clicking #download fetches a pretend model file. While it downloads, keep #bar (a <progress> with max="100") and #pct (text such as 40%) up to date from the bytes received and the Content-Length header. When it ends, put Ready in #status.

Intermediate
3

Label Text in a Worker, With Cancel

A worker script is already written: it receives text and, after a moment, answers returns or other. Clicking #run should show Working… in #result, send the text from #text to a worker and show the label when it answers.

Intermediate
4

Survive a Full Disk

Three buttons try to save a model to a pretend cache with a 500-byte quota using saveToCache(kind, size). Write save(kind, size) so #status says: Saved for next time on success, Not enough space. Running without saving. for a QuotaExceededError, and Something went wrong for any other error.

Intermediate
5

Device First, Then the Server

Build the hybrid. Clicking #ask should first try askDevice(question, supported), where supported is the checkbox #supported. If it returns with confidence of at least 0.5, show its text in #answer and on this device in #source.

Advanced