Topic 0.5
Your First Real Embeddings
In one line
An embedding model turns a text into a list of numbers so that texts with similar meaning get similar numbers. This topic runs a real one in your browser on the question that word matching got wrong, and explains how every example and lab in the course works.
Think of it like this
A library that shelves books by subject, not by title. Two books on bike repair sit side by side even if their titles share no word. To find books like the one in your hand, you look at its neighbours on the shelf. An embedding model gives every text a place on a shelf with hundreds of directions, so that texts about the same thing end up close together.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Embedding
- A list of numbers a model produces for a text, placed so that similar meanings get similar lists.
- Vector
- A list of numbers. An embedding is a vector.
- Dimensions
- How many numbers are in the vector. The lab's model uses 384.
- Embedding model
- A model whose output is a vector for the input text, not more text.
- Cosine similarity
- A score for two vectors: near 1 when they point the same way, near 0 when unrelated.
- Semantic search
- Search by meaning: rank texts by how close their embeddings are to the question's embedding.
- Toy embedder
- A hand-made stand-in used in this course's runnable examples, so every line can be read and all three languages agree.
Step by step
01Step 1: a toy you can read
The runnable example defines four directions: account, money, time and safety. A table says which words point where: password, login and forgot all point along *account*. To embed a text, count its words per direction and scale the result to length 1.
MEANING = {"password": 0, "login": 0, "forgot": 0, "refunded": 1, "hours": 2, "helmet": 3}
def embed(text):
vector = [0.0, 0.0, 0.0, 0.0] # account, money, time, safety
for word in tokenize(text):
if word in MEANING:
vector[MEANING[word]] += 1.0
length = math.sqrt(sum(x * x for x in vector))
return vector if length == 0 else [x / length for x in vector]02Step 2: the toy finds the paraphrase
Run it. The login question becomes [1, 0, 0, 0]: all *account*. The password passage is mostly *account* too, so it scores about 0.96 and every other passage scores 0. The toy succeeds because we told it that login and password belong together. That knowledge is the whole trick.
03Step 3: where the toy stops
Ask the toy about "sign in" or "credentials" and it knows nothing: those words are not in its table. A real embedding model was trained on an enormous number of sentence pairs and has, in effect, a table for the whole language, with hundreds of directions that nobody named. Phase 2 explains how it learns them.
04Step 4: the same question, with a real model
Open the first search lab and press Search. Your browser downloads the model once, turns the question and five passages into 384 numbers each, and ranks the passages. Look at the Vectors tab to see the first few numbers of each, and at the Similarity grid to see every pair compared.
05Step 5: compare with keyword search
In the lab, change Rank by to Keyword (BM25). BM25 is the proper version of word matching that Phase 1 builds. It repeats Topic 0.4's mistake: the refunds passage scores about 2.63 (it contains in and get), the password passage about 1.26 (only forgot), so refunds comes first. Now ask "I can't sign in": the only shared word left is in, keyword search returns the refunds passage alone, and the vector ranking still puts password reset first, at about 0.40.
06Step 6: read scores as a ranking, not as certainty
In the second lab, ask about scooters, which the handbook never mentions. The best passage still gets a score, about 0.38 for the helmet passage. Compare: the answerable login question's best score was about 0.47, and "I can't sign in" scored about 0.40. The gap between "found it" and "nothing relevant" is small and depends on the model. Deciding when to refuse needs care and measurement (Topic 8.4).
Try it yourself
- 1
Teach the toy a word
In the runnable example, ask
"I lost my credentials". The toy scores every passage 0. Add"credentials": 0to the table and run again. You just did by hand, for one word, what training does for a whole language. - 2
Read a real vector
In the first lab, open the Vectors tab after a search. Each text became 384 numbers; the first 8 are shown, with the number of tokens the model read. Change one word in a passage, search again, and watch which numbers move.
- 3
Break keyword search, not vector search
In the first lab, ask
"I can't sign in"with Rank by set to Keyword (BM25), then to Vector. Which single word lets keyword search find anything, and is that passage useful? What does the vector ranking put first, and with what score? Then try Hybrid (RRF): it mixes both rankings, and Phase 6 explains why it can tie. - 4
Use the grid
Open the Similarity grid. Which two *passages* are most similar to each other? Does that match your own reading of them?
Code & diagrams
A toy: its four directions and its word table were written by hand. A real model has hundreds of unnamed directions learned from data. Real numbers are in the lab below.
Expected output
Directions: account, money, time, safety
Question: I forgot my login, how do I get back in?
vector: [1.000, 0.000, 0.000, 0.000]
0.962 D3
0.000 D1
0.000 D2
0.000 D4
0.000 D8
Question: When do you close?
vector: [0.000, 0.000, 1.000, 0.000]
1.000 D2
0.447 D1
0.316 D8
0.275 D3
0.000 D4Try: Press Search and read the Cosine column: the password passage wins although it shares only one word with the question. Then set Rank by to Keyword (BM25) and see which passage keyword search prefers, and why. Finally try the question "I can't sign in" with both rankers.
Goals
- The password-reset passage is the best match
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Try: Search for scooters, which no passage mentions, and note the best cosine score. Then solve the goal: write a question about helmets that scores at least 0.60 against the helmet passage. Which wording gets closest to 1?
Goals
- The helmet passage scores at least 0.60 for your question
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to scale the vectors
In the Python tab of the toy embedder, make embed return the raw counts: replace its last line with return vector. Run.
Myth vs fact
Myth
A cosine score of 0.47 means the model is 47 percent confident.
Fact
It is a similarity, not a probability. Only compare scores from the same model with each other. One model's 0.47 can be a strong match and another's a weak one.
Myth
Each of the 384 numbers stands for a concept like "money" or "time".
Fact
Only in the toy. In a real model the directions are learned, unnamed and mixed. The meaning is in how whole vectors compare.
Myth
Embedding models and chat models are the same thing.
Fact
Both are neural networks that read text, but a chat model outputs more text and an embedding model outputs one vector. They are separate models, usually with separate prices and limits.
Myth
Vector search understands the question.
Fact
It finds text that is *similar* to the question. Similar is often relevant, but "Do you rent scooters?" is similar to passages about renting bikes, none of which answers it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
The model in the lab is small and several years old, chosen because it downloads quickly and runs anywhere. Stronger models rank better and separate relevant from irrelevant more widely, but the mechanics are identical. Topic 2.7 covers choosing one.
- ▸
Embedding models have an input limit. The lab's model reads at most 512 tokens of a text and silently ignores the rest: text past the limit does not change the vector at all. This is one reason documents are cut into chunks (Phase 4), and Topic 2.6 shows it in a lab.
- ▸
The lab embeds texts one at a time. With this quantised model, embedding texts together in a padded batch changes the numbers slightly (about 0.01 in a similarity). Batch size should never change a ranking you depend on; when it can, fix the batching in tests.
- ▸
Running a small embedding model in the browser is a real architecture, not only a teaching trick: the text never leaves the device, there is no per-request cost, and it works offline. The trade is a one-time download and a weaker model.
Remember this
- 1
Why this exists. Topic 0.4 ended with a search that could not see that "forgot my login" and "reset your password" mean nearly the same thing. Word matching compares spellings. We need to compare meanings, and a computer can only compare numbers. So we need a function from text to numbers that keeps meaning: similar meaning in, similar numbers out. That function is an embedding model.
- 2
What an embedding is. An embedding is a list of numbers, called a vector, that a model produces for a piece of text. The model in this course's lab produces 384 numbers for any text, whether it is one word or a paragraph. The count of numbers is the vector's dimensions. No single number means anything you could name. What matters is how two vectors compare.
- 3
Comparing two vectors. The usual measure is cosine similarity: a number that is close to 1 when two vectors point the same way, near 0 when they are unrelated, and below 0 when they point in opposite directions. To search, embed the question, embed every passage, and rank passages by their cosine similarity to the question. LLM Engineering Topic 1.3 introduces this in one topic; Phase 2 of this course takes it apart properly.
- 4
How this course runs code. Every runnable example has Python, Java and JavaScript tabs that print exactly the same output. Three different language runtimes cannot share one real embedding model, and a real model's 384 numbers would be hard to follow anyway. So runnable examples use a toy embedder: a small table, written in the example, that says which words point in which direction. It is honest about being a toy, and it lets you read every line of the mechanism.
- 5
The search lab: a real model, in your browser. Where real vectors teach something a toy cannot, the page has a search lab. It runs a real open embedding model (
all-MiniLM-L6-v2, 384 dimensions, English) inside your browser. The first Search downloads the model once, about 23 MB; after that it is instant, free, needs no sign-in, and your text never leaves the page. You edit the question and the passages, press Search, and see every passage's real score. - 6
The answer step: a real chat model. Some labs add the last step of RAG: the best passages and your question are sent to a real, free hosted language model, and you see its answer. That part goes through this site's server (the key stays there), needs a free sign-in, and has a daily limit, like the lab in LLM Engineering.
- 7
What the scores mean, and don't. In the first lab below, the password-reset passage scores about 0.47 for the login question and every other passage scores below 0.12. The right passage wins clearly, with no word in common except "forgot". But 0.47 is not "47 percent sure". Cosine scores are only meaningful compared with each other, for the same model. A different model would give different numbers for the same texts. The second lab makes this concrete.
Explain it without notes
What is an embedding, and what property makes it useful for search?
Why do the runnable examples use a toy embedder while the labs use a real model?
In the lab, the best passage for the login question scores about 0.47. A colleague says "that's below 50 percent, so it's a bad match". What do you tell them?
The scooter question gets a best score of about 0.38 although nothing in the handbook is about scooters. Why, and what does that mean for building a product?
Practice
Write cosine(a, b) for vectors that are not already length 1 (divide by both lengths). Print the similarity of [3, 0, 1, 0] with [1, 0, 0, 0] and with [0, 2, 0, 0], to three decimals.
Using the toy idea, write embed(text, meaning, size) and rank three short texts for the question "is it open on sunday" with the table {"open": 0, "sunday": 0, "hours": 0, "price": 1, "costs": 1}.
Trade-offs
- ↔
Toy embedder: readable, identical in three languages and instant, but it only knows the words you list.
- ↔
Real model in the browser: real vectors, free and private, but a one-time download and a small model.
- ↔
A hosted embedding API: the strongest models and no download, but a key, a cost per token, rate limits, and your text leaves your system.
- ↔
Vector search: finds paraphrases, but returns something for every question, including ones with no answer.
Done when you can
Done when you can explain an embedding as "text to numbers, with similar meanings close together".
Done when you can say what cosine similarity is close to for similar, unrelated and opposite vectors.
Done when you have run both search labs and can state the real scores you saw for the login and scooter questions.
Done when you can explain why a similarity score is not a confidence.
Done when you can say which parts of this course run in your browser and which call a server.