PHASE 12Advanced ~26 min· topic 2 of 8

Topic 12.2

Transformers.js and ONNX Runtime Web

In one line

Transformers.js lets you run a trained model in the browser with one call, pipeline(task, model), and runs it on ONNX Runtime Web. You choose a task, a model and options, you wait for the files to load, and then you call the result like a function.

Think of it like this

A coffee machine with pods. You do not grind beans or heat water yourself. You choose a pod (the model), press one button (the pipeline) and get a cup (the result). The first time, you have to buy and carry the pod home (the download). After that it is one press. A pipeline is the button: it hides the tokenizer, the model file and the maths behind one function.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pipeline: A ready-made function for one task. You create it once with pipeline(task, model) and call it with your input.
Task: What you want done, such as classifying text or turning text into numbers.
ONNX: An open file format for trained models. One ONNX file can run on different runtimes.
ONNX Runtime Web: The engine that runs ONNX models in the browser, on WebAssembly or WebGPU.
Tokenizer: The part that cuts text into the pieces (tokens) a model understands.
dtype: The precision of the model weights, such as fp32, fp16, q8 or q4. Lower is smaller and faster but less exact.
Quantised: Stored with fewer bits per number to make the file smaller. q8 and q4 are quantised.
progress_callback: A function the library calls while files download, so you can show a progress bar.
Revision: A specific saved version of a model on the Hub. Pinning it keeps results the same for everyone.

Step by step

01The real code, which does not run in this sandbox

This is the whole first task with the real library. It needs the network to download the model, so it cannot run in the sandboxed examples on this page. Everything after the first line is the same shape in any task.

classify.js (real library, not live)whole filejs

import { pipeline } from "@huggingface/transformers";

// 1. Slow, once: downloads the model files and prepares the runtime
const classify = await pipeline("sentiment-analysis");

// 2. Fast, many times: just call it
const out = await classify("I love transformers!");
// [{ label: 'POSITIVE', score: 0.999817686 }]   (the library's own docs example)

02Choose the model, the device and the precision

Pass a model name as the second argument and options as the third. The options below ask for the graphics chip and a 4-bit model. Not every model offers every dtype, and not every device has a GPU, so a real app picks these after the checks in Topic 12.3.

options.js (real library, not live)whole filejs

const pipe = await pipeline(
  "sentiment-analysis",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
  {
    device: "webgpu", // default in the browser is WebAssembly on the CPU
    dtype: "q4",      // fp32 | fp16 | q8 | q4, if the model offers it
    progress_callback: (p) => {
      if (p.status === "progress") console.log(p.file, p.loaded, "of", p.total);
    },
  },
);

03The same shape, runnable here

The first live example defines a stand-in pipeline() with the same call shape: a task and a model go in, a function comes out, and a progress_callback is called per file. It is not the real library and the "model" is a one-line rule. Read it for the structure: the task table, the load that happens once, and the checks on the input.

Three things to see in the output. The progress lines appear once, because the second request reuses the first pipeline. An empty string goes through as an ordinary input. A number is refused with a TypeError, and an unknown task is refused with a message that lists the supported ones.

04A failed load must not stick

The second example is the bug that bites everyone once. If you store the promise from the first load and that load fails (a dropped connection), the stored promise is a rejected one, and every later call gets the same failure forever, even after the network is back.

The fix is two lines: when the promise rejects, clear the variable, so the next call starts a new load. This site's own embedding worker (public/runner/embed-worker.js) does exactly this.

A failed load must not stickdiagram

Rendering diagram…

05Settings you may need later

The library has a global env object. Names verified against its documentation: env.allowRemoteModels (default true), env.localModelPath (where local models are read from), env.useBrowserCache (uses the Cache API, default true when available), env.cacheKey (default "transformers-cache") and env.useWasmCache (also caches the runtime's own WebAssembly files, default true when a cache is available). In a browser, env.allowLocalModels is documented as false by default and env.localModelPath as the /models/ URL path, so to host files yourself you set allowLocalModels to true as well. Check your installed version's behaviour once with the browser's network panel. Topic 12.4 uses the cache ones; one use of localModelPath is hosting the model files on your own server instead of the Hub.

env.js (real library, not live)whole filejs

import { env } from "@huggingface/transformers";

env.allowRemoteModels = false;   // never call the Hub
env.allowLocalModels = true;     // documented default is false in a browser, so set it when you host the files
env.localModelPath = "/models/"; // read model files from your own server

Try it yourself

1
Add a task
In the first example add a third task to TASKS, for example "language-guess", that returns [{ label: /[\u0900-\u097F]/.test(text) ? "hindi" : "other", score: 0.9 }]. Call it with a Hindi sentence.
The pattern is the point: a task name picks a function, and the pipeline returns that function. A real library does the same with real models behind each name.
2
Load two at once
In the first example replace the two awaits with const [a, b] = await Promise.all([getClassifier(), getClassifier()]);.
Check that downloads is still 1. Because the code stores the promise, not the finished pipeline, two callers at the same moment share one load. Without that, two parts of a page would each start a download.

Code & diagrams

A pipeline-shaped stand-in: load once, reuse, check the input

The pipeline() here is a stand-in with the real library's call shape, not the library itself. Real use: import { pipeline } from "@huggingface/transformers" (not available in this sandbox).

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

file ready: config.json
file ready: model.onnx
[{"label":"RETURNS","score":0.9}]
[{"label":"OTHER","score":0.9}]
same pipeline reused: true | downloads: 1
"" -> [{"label":"OTHER","score":0.9}]
42 -> TypeError: Expected a string, got number
Unsupported task: sentiment-analysis. Supported: text-classification, feature-extraction

Do not remember a failed load

The first load attempt fails and the second works. The wrong version stores the rejected promise, so it can never recover. The right version clears it.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

wrong, call 1: Failed to fetch model.onnx (network)
wrong, call 2: Failed to fetch model.onnx (network)
right, call 1: Failed to fetch model.onnx (network)
right, call 2: intent-model
loader ran 2 times in the right version

Loading the pipeline inside a Web Worker (real library, not live)js

The shape of this site's own embedding worker. It loads on the first message, reuses the pipeline, and clears a failed load so it can be retried.

// worker.js
let ready = null;
function load() {
  if (!ready) {
    ready = import("@huggingface/transformers")
      .then(({ pipeline }) => pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2", { dtype: "q8", device: "wasm" }));
    ready.catch(() => { ready = null; }); // a failed download can be retried
  }
  return ready;
}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Create the pipeline every time

In the first example change classifier ??= pipeline( to classifier = pipeline( so a new pipeline is made on every call.

terminal

$ The last lines of the output

── what you'll see ──

file ready: config.json

file ready: model.onnx

...

same pipeline reused: false | downloads: 2

Break #2

Remember the failure

In the second example, delete the line goodCache.catch(() => { goodCache = null; });.

terminal

$ The right version's lines

── what you'll see ──

right, call 1: Failed to fetch model.onnx (network)

right, call 2: Failed to fetch model.onnx (network)

loader ran 1 times in the right version

Myth vs fact

Myth

Transformers.js is a model.

Fact

It is a library. Models are separate files, published on the Hugging Face Hub, that the library downloads and runs.

Myth

pipeline() is fast to call.

Fact

Creating it downloads and prepares the model, which can take a long time. Create once, then call the returned function many times.

Myth

Smaller dtype is always better for the user.

Fact

q4 and q8 files are smaller and load faster, but can be less accurate. Test the accuracy on your own inputs and check the model card for what is offered.

Myth

You need a server to use Hugging Face models.

Fact

The browser can download the files and run them itself. You may still host the files yourself with env.localModelPath for control and reliability.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Pin the model. Use the revision option with a specific commit of the model repository, as this site's embedding worker does, so a change on the Hub cannot change your results or break your cache. Likewise pin the library version in the import URL or package.json.
▸
The ONNX Runtime settings are separate from the pipeline settings. ort.env.wasm.numThreads controls threads (default 0, which means the runtime chooses; multiple threads need the page to be cross-origin isolated, Topic 12.3), and ort.env.wasm.wasmPaths can point at where the runtime's WebAssembly files are served from. The runtime's script and its WebAssembly files must come from the same version, or start-up fails.
▸
Treat model hosting like asset hosting. The default Hub download is convenient, but a production app often serves model files from its own origin or CDN, so it controls availability, caching headers and what a corporate network allows.

Remember this

1
The problem this solves: A trained model is a file of numbers plus a lot of surrounding code: turning text into tokens, running the numbers, and turning the output back into labels. Writing that yourself is a project. Transformers.js is a JavaScript library that does all of it, runs in the browser, and loads models published on the Hugging Face Hub. This topic shows the shape of the API and what happens at each step.
2
The pieces. *Transformers.js* is the library you call (npm package @huggingface/transformers, version 4.3.1 when this was written). It runs models through *ONNX Runtime Web* (npm package onnxruntime-web, version 1.30.0 when this was written). ONNX is an open file format for trained models, so one model file can run in many places. The runtime is the engine that executes it, on the CPU through WebAssembly or on the graphics chip through WebGPU (Topic 12.3). Check the versions when you start: both move fast.
3
A pipeline takes a task, a model and options. The task says what you want done, for example "sentiment-analysis", "feature-extraction" (turn text into numbers, Topic 12.6), "automatic-speech-recognition" or "image-classification". The model is a name on the Hub such as "Xenova/distilbert-base-uncased-finetuned-sst-2-english"; without one the library picks a default for the task. pipeline(...) returns a promise of a function. You await it once (this is the slow part: files load), then call it with your input as often as you like.
4
What gets downloaded. A model is several files: a configuration, files for the tokenizer (the part that cuts text into pieces the model understands), and the model weights in ONNX format. Weights come in different precisions, set with the dtype option: the documentation lists common values such as "fp32", "fp16", "q8" and "q4", and says the list available depends on the model (the library can probe a model repository for what it offers with ModelRegistry.get_available_dtypes()). When you do not set dtype, the library's source picks "q8" for WebAssembly and "fp32" for other devices such as WebGPU. Lower precision means a smaller file and faster loading, at some cost in accuracy. Never trust a number you remember: check the model card for the file sizes.
5
Options you will use. device: "webgpu" asks for the graphics chip; the default in the browser is WebAssembly on the CPU. dtype picks the precision. progress_callback is a function called while files download; the object it receives has a status, and for download progress status is "progress" with file, loaded and total (this site's own embedding worker uses exactly these). A revision option pins the exact model version so every user gets the same files.
6
Load once, reuse. Loading is the expensive step, so keep the pipeline in a variable and reuse it. Keep the *promise* too, so two parts of the page that ask at the same moment share one load. And if loading fails, forget the failed promise so the next attempt can try again. The two live examples show exactly this with a stand-in that has the same call shape.

Explain it without notes

What does pipeline(task, model) do, and why is it called once?

What are device and dtype, and what do they trade off?

Why must a failed model load be cleared from the stored variable?

What is the relationship between Transformers.js and ONNX Runtime Web?

Practice

Write makeLoader(load) that takes an async function and returns a get() that calls load once, shares the promise between simultaneous callers, and retries after a failure. Test it with a loader that fails once.

Write the progress_callback for the real library that sums the bytes over all files into one percentage and logs it only when the percentage changes.

Trade-offs

↔
Using pipeline() is fast to write and hides the details, but you give up control of tokenizing and batching unless you drop to lower-level classes. For most product features the pipeline is enough.
↔
Hosting model files yourself costs bandwidth and care but gives you control. The Hub is free and convenient, but you depend on its availability and on the user's network reaching it.

Done when you can

I can write the shape of a pipeline(task, model, options) call and explain each argument.
I can explain what ONNX and ONNX Runtime Web are and how they relate to Transformers.js.
I can say what device, dtype and progress_callback do.
I can load a pipeline once, share it between callers and recover from a failed load.
I can say why a model should be pinned to a revision.

Command Palette

Words you'll meet

Step by step

01The real code, which does not run in this sandbox

02Choose the model, the device and the precision

03The same shape, runnable here

04A failed load must not stick

05Settings you may need later

Try it yourself

Add a task

Load two at once

Code & diagrams

Break it on purpose

Create the pipeline every time

Remember the failure

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can