PHASE 12Advanced ~25 min· topic 6 of 8

Topic 12.6

Embeddings and Semantic Search On-Device

In one line

An embedding turns a piece of text into a list of numbers so that texts with similar meaning get similar numbers. With a small embedding model in the browser you can search help articles by meaning, as the user types, with no server and no text leaving the device.

Think of it like this

A library where books are placed on shelves by topic, not by title. If you stand near the cooking books and reach left, you find baking. The position tells you what a book is about. An embedding is a position on a huge map of meaning: texts that mean similar things land close together, even if they share no words. Searching is walking to the spot where your question lands and looking at what is nearby.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Embedding: A list of numbers that represents the meaning of a text. Similar meanings give similar lists.
Vector: A list of numbers. An embedding is a vector, for example with 384 numbers.
Dimensions: How many numbers are in the vector. Fixed by the model.
Cosine similarity: A score for how similar two vectors are by direction: 1 is the same, 0 unrelated.
Normalise: Scale a vector to length 1. Then cosine similarity is a plain dot product.
Index: The stored list of document vectors you search through.
Semantic search: Search by meaning, rather than by matching words.
Debounce: Wait until the user stops typing for a moment before running the search.
Stale result: An answer to an older query that arrives after a newer one has started. It must be ignored.

Step by step

01How a real model is called

This is the call this site's own embedding worker makes, shown as a real code block. It cannot run in the sandbox here (no network to fetch the model), but you can run the real thing in the RAG course. pooling: "mean" turns one vector per token into one vector per text, and normalize: true gives it length 1.

embed.js (real library, not live)whole filejs

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2", {
  dtype: "q8",
  device: "wasm",
});
const out = await extractor("Can I get my money back?", { pooling: "mean", normalize: true });
const vector = Array.from(out.data); // 384 numbers, length 1

02Index once, search many times

In the first live example the four FAQ answers are embedded when the page loads, into index. Searching embeds only the query, compares it with each stored vector, drops scores below minScore, sorts and keeps two.

Read the scores in the output. "return an item" scores highest against the returns answer. The scores of the other results are not zero, because the toy embedding has random hash collisions, and "how long does shipping take" is almost a tie (0.47 against 0.46). That is why real search UIs apply a minimum score and sometimes a margin between first and second place, and why you should not show a result that only barely wins as if it were certain.

03Edge cases

An empty query, an emoji and a Hindi question all produce a zero vector in the toy model. The function checks for it and returns [], so the screen can show "no match, try asking in other words" and offer the server fallback (Topic 12.7). A real multilingual model would handle Hindi; an English-only model would give meaningless scores. Know which language your model was made for and test with your users' languages.

04Typing, debouncing and stale answers

The second example has three parts. No guard: two searches start for "ret" and then "return"; "ret" is slower, so its answer arrives last and replaces the right one. A request id: the late answer is ignored, with a log line. Debounce: five calls in a row start only one search.

Together they are the standard pattern: debounce to start fewer searches, and an id to make the ones that do start safe.

Typing, debouncing and stale answersdiagram

Rendering diagram…

05What to put on the screen

Empty box: show popular articles. Searching: keep the old results visible and dim them, rather than flashing a spinner on every keystroke. No results: say so and offer a way out (ask the assistant). While the model is still loading the first time: say "Getting search ready" with the progress from Topic 12.4, and let the user type; run the search when the model is ready. Mark results with how they were found if you also have plain word search.

Try it yourself

1
Raise the minimum score
In the first example change the default minScore = 0.3 to 0.5 and run it. Fewer results survive: "return an item" and "masala chai" keep only their best result, and "how long does shipping take" (0.47 and 0.46) returns nothing at all.
Trace the effect of the threshold on your own questions. This is the same trade-off as the confidence threshold in Topic 12.1.
2
Poison the order
In the second example swap the two latencies: { ret: 10, return: 60 }. Run it.
Now the answers arrive in the right order, so "no guard" shows the right result by luck. Bugs that depend on timing hide on fast networks and appear on slow ones.

Code & diagrams

Index once, search by vector, handle empty and non-Latin input

This toy embedding matches spelling pieces, not meaning. A real model (see the RAG course) would match "money back" to the returns answer. The code around the embedding is the same.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"return an item" -> ["0.62 You can return any uno","0.44 Our masala chai is bla"]
"how long does shipping take" -> ["0.47 Orders ship in 1 to 2 ","0.46 You can return any uno"]
"masala chai" -> ["0.51 Our masala chai is bla","0.39 You can return any uno"]
"" -> []
"👍" -> []
"मेरा ऑर्डर" -> []

Search as you type: stale answers and debounce

slowSearch is a stand-in whose speed we control: the shorter query is slower, so the answers arrive in the wrong order, as they can in real life.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

no guard shows: results for ret
ignored late answer: results for ret
with guard shows: results for return
search starts for: return
searches started: 1

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the stale-answer guard

In the second example, change if (mine !== latest) { to if (false) {.

terminal

$ The with-guard line of the output

── what you'll see ──

with guard shows: results for ret

Myth vs fact

Myth

Embeddings understand text the way a person does.

Fact

They place text on a map learned from lots of examples. It is very useful, but it can put unrelated things close together and miss subtle differences, so a minimum score and a fallback are needed.

Myth

Any embedding model can be used for any language.

Fact

Models are trained on specific languages. An English-only model gives poor results for other languages. Check the model card.

Myth

You can mix vectors from different models.

Fact

Vectors from different models (or versions) live on different maps and cannot be compared. Store the model name and version with the index and rebuild when they change.

Myth

Semantic search replaces keyword search.

Fact

Each is better at something. Exact names, codes and order numbers ("ORD-1042") are found best by keyword search. Many products combine both.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Linear scan is fine for thousands. Comparing a query with 5 000 stored vectors of 384 numbers is about two million multiply-adds, which a modern device does in a few milliseconds. Special index structures matter when you have hundreds of thousands. Measure before you add complexity.
▸
Store vectors compactly. A vector of 384 numbers stored as 32-bit floats is 1 536 bytes, so 5 000 documents is about 7.7 MB before overhead. Keep them in a typed array (Float32Array) rather than arrays of arrays, and transfer them (Topic 12.5) rather than copying.
▸
Embed the questions of tomorrow, today. Precompute document vectors at build time and ship them as a file, so users download vectors (small) rather than embedding thousands of texts on their device (slow). Only the query is embedded in the browser.
▸
Guard privacy: a search box that embeds locally is private only if you do not also log the query. If analytics receive every keystroke, the on-device model changed nothing.

Remember this

1
The problem this solves: Ordinary search matches words. A customer who types "money back" does not find the article called "Returns and refunds", because no word matches. Search by meaning finds it. Doing this on the device means the search box works instantly, offline, and the customer's question is never sent anywhere. This topic builds the whole mechanism; the RAG course (/rag) shows what to do with it next, and runs a real embedding model in your browser.
2
An embedding is a vector: a list of numbers, for example 384 of them. An embedding model is a small neural network that reads text and outputs such a list. Texts that are close in meaning get vectors that point in similar directions. You do not read the numbers; you compare them. The model is trained so that this works for general language, and a model made for one language may do badly on another.
3
Cosine similarity is the comparison. It measures the angle between two vectors: 1 means the same direction, 0 means unrelated, and negative means opposite. If both vectors are scaled to length 1 (normalised), cosine similarity equals the dot product: multiply the numbers pairwise and add. That is a fast loop, so searching a few thousand items on the device is quick.
4
The real setup in the browser. This site's RAG course does it with Transformers.js: a feature-extraction pipeline with the model Xenova/all-MiniLM-L6-v2, dtype: "q8", device: "wasm", and pooling: "mean", normalize: true when calling it. Its source says the vectors have 384 dimensions and the model file is about 23 MB, downloaded once and kept in the browser's cache, running inside a Web Worker. Everything in Topics 12.2 to 12.5 is in that worker. The code block below shows its call.
5
Two phases. *Indexing:* embed every document once (titles, answers, anything searchable) and keep the vectors. Do this ahead of time or when the page first loads, and save the result (IndexedDB or a file), because embedding thousands of texts takes real time. *Searching:* embed the query with the same model, compare it with every stored vector, sort by score, keep the top few. Vectors from different models, or different versions of a model, cannot be compared: store the model name and version with the vectors and rebuild when it changes.
6
A stand-in you can read. The first live example builds a toy embedding that counts three-letter pieces of the text into 64 buckets. It is not a neural network and it does not understand meaning: it matches spelling. Its job is to show the code around it: index once, embed the query, take the dot product, filter by a minimum score, sort, keep two. Swap in the real model and that code does not change. Empty text, emoji and Hindi give a vector of all zeros, and the code returns an empty list rather than a nonsense ranking.
7
Search as you type adds three problems. Each keystroke would start an embedding, so debounce: wait until the user pauses. Answers can come back out of order, so give each search an id and ignore any answer that is not the newest. And an empty or one-letter box should show the default list, not a search result. The second live example shows all three: an unguarded search shows the old query's results, a guarded one ignores them, and five keystrokes become one search.

Explain it without notes

What is an embedding, and how do you compare two of them?

Why must the document vectors and the query vector come from the same model?

How do you handle empty, emoji or unsupported-language input in a search box?

What problems does search-as-you-type add, and how do you solve them?

Practice

Add a margin rule to the first example's search: if the best score is less than 0.05 above the second, return [] with a log line "too close to call".

Store the index as one Float32Array of length FAQ.length * DIMS and write scoreAll(q) that returns an array of scores without creating an array per document.

Trade-offs

↔
Embedding the corpus at build time makes the page light and fast, but the index goes stale when content changes and must be rebuilt with the same model version. Embedding in the browser keeps the index fresh but costs start-up time on every device.
↔
Semantic search finds meaning but can surprise users with odd matches; keyword search is predictable but misses synonyms. Combining them costs more code and gives the best results.

Done when you can

I can explain embeddings, vectors and cosine similarity in plain words.
I can describe how a real embedding model is called in the browser, and what it needs.
I can build an index once and search it with a query vector.
I can handle empty, emoji and unsupported-language input without a bad ranking.
I can protect search-as-you-type with debounce and a stale-answer guard.

Command Palette

Words you'll meet

Step by step

01How a real model is called

02Index once, search many times

03Edge cases

04Typing, debouncing and stale answers

05What to put on the screen

Try it yourself

Raise the minimum score

Poison the order

Code & diagrams

Break it on purpose

Remove the stale-answer guard

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can