Topic 1.3
Embeddings: Meaning as Numbers
In one line
An embedding turns text into a list of numbers so that similar meanings get nearby lists. Cosine similarity measures how close two lists point, which is the base of semantic search and RAG.
Think of it like this
A map. Two cafes that are close on the map are close in real life. An embedding is a map of meaning: every word or sentence gets a position, and things with similar meaning get nearby positions. The map has hundreds or thousands of directions instead of two, but the idea is the same.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Embedding
- A list of numbers that stands for the meaning of a piece of text.
- Vector
- A list of numbers. Its length is the number of dimensions.
- Dimension
- One position in the list. Real embeddings have hundreds to thousands.
- Dot product
- Multiply matching positions of two vectors and add the results.
- Cosine similarity
- The dot product divided by both vector lengths. It compares direction only, from -1 to 1.
- Embedding model
- A model trained to turn a text into one vector for comparing texts.
- Normalised vector
- A vector scaled to length 1. For these, cosine similarity equals the dot product.
Step by step
01Step 1: the search problem
You have 1,000 help articles and a user types "my phone will not switch on". An article titled "Device fails to power up" shares no word with the query. Keyword search misses it. We need a comparison that sees meaning.
02Step 2: give each text a position
Make a table of vectors. The example uses three numbers per word. cat, dog and kitten are given similar numbers, and car and truck other similar numbers. In real systems a trained embedding model produces these numbers for you.
03Step 3: the dot product
dot(cat, dog) = 0.9*0.85 + 0.8*0.75 + 0.1*0.2 = 1.385. For cat and car it is only 0.345. Bigger dot product, more alike. But look at the last lines of the output: doubling a vector doubles its dot product while its meaning is unchanged, so raw dot products can be misleading when lengths differ.
cat, dog = [0.9, 0.8, 0.1], [0.85, 0.75, 0.2]
print(sum(x * y for x, y in zip(cat, dog)))04Step 4: cosine similarity fixes the length problem
Divide by the two lengths. Now cosine(cat, doubled cat) is exactly 1. The ranking in the output puts dog (0.9958) and kitten (0.9923) next to cat, banana in the middle (0.7503), and car and truck far away (below 0.3).
05Step 5: use it
Real use: embed every document once and store the vectors, embed each query at request time, compute similarities (or ask a vector database to find the nearest), and return the top few. The most common bugs are mixing two embedding models and forgetting that a model update changes every vector, so old stored vectors no longer match new queries.
Try it yourself
- 1
Add a word
Add
"puppy": [0.8, 0.7, 0.25]and run. Where does it rank among the neighbours ofcat? Then add"tiger": [0.95, 0.5, 0.1]and guess its rank first. - 2
Dot or cosine?
Add
"big cat": [1.8, 1.6, 0.2](twicecat) and compare. Which score ranks it first, and which ranks it equal tocat? - 3
Opposite direction
Add
"anti": [-0.9, -0.8, -0.1]and computecosine(cat, anti). You should get -1.0000.
Code & diagrams
Real embeddings come from an embedding model and have far more numbers, but the maths is identical.
Expected output
dot(cat, dog) = 1.3850
dot(cat, car) = 0.3450
closest to cat by cosine:
dog 0.9958
kitten 0.9923
banana 0.7503
car 0.2926
truck 0.2750
dot(cat, doubled cat) = 2.9200
cosine(cat, doubled cat) = 1.0000Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Compare vectors from different models
Pretend a second embedding model produced [0.1, 0.8, 0.9] for the word cat (its numbers mean other things). At the end of the Python version add print(f"cosine(cat, other model vector) = {cosine(cat, [0.1, 0.8, 0.9]):.4f}").
Myth vs fact
Myth
Each number in an embedding means something I can read, like "animal-ness".
Fact
Dimensions are learned and mixed. Only the distances and angles between whole vectors carry meaning.
Myth
A higher dot product always means more similar.
Fact
It also grows with vector length. Use cosine similarity, or make sure the vectors are normalised to length 1.
Myth
Embeddings understand facts.
Fact
They capture similarity of use and topic. "The bridge is open" and "The bridge is closed" are very close, though they say opposite things.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Many embedding APIs return vectors already normalised to length 1. Then cosine similarity equals the dot product, which is faster, and vector databases use that.
- ▸
Searching millions of vectors one by one is too slow, so vector databases use approximate nearest neighbour indexes (such as HNSW) that trade a little recall for large speed-ups. The RAG course covers them.
- ▸
Embedding quality is domain-specific. Test candidate models on your own queries with a small labelled set (Phase 9) rather than reading leaderboards, and record the model name and version next to stored vectors.
Remember this
- 1
Why this exists. Computers compare numbers well and meaning badly. A search for
carshould find a page aboutautomobileeven though the letters differ, and plain string search cannot do that. If every text becomes a position on a meaning map, then "similar meaning" becomes "small distance", which is easy to compute and to index. - 2
A vector is just a list of numbers.
[0.9, 0.8, 0.1]is a vector with three dimensions. In the example,cat,dogandkittenhave similar vectors,carandtruckhave similar vectors, and the two groups are far apart. Real embeddings are learned from data so that this property emerges; nobody writes the numbers by hand. - 3
Dot product and cosine similarity. The dot product multiplies matching positions and adds them up. It is larger when two vectors point the same way, but it also grows with vector length (the example doubles
catand its dot product doubles). Cosine similarity divides the dot product by both lengths, so it measures only the angle: 1 means the same direction, 0 means at right angles (no shared direction, so roughly unrelated), -1 means opposite. It ignores length, which is usually what you want for meaning. A vector scaled to length 1 is called normalised. - 4
Two different things share the name. Inside every model there is an input embedding table: one learned vector per token id, the first step the model does with its input. Separately, there are embedding models, trained to turn a whole sentence or document into one vector for comparing texts. You call the second kind through an API (a dedicated endpoint). Do not mix the two.
- 5
Where you use them. Semantic search (embed the documents once, embed the query, return the nearest), RAG (retrieve the passages nearest to the question and put them in the prompt), clustering and deduplication, and recommendations. Phase 8 builds this fully. Two rules matter now: use the same embedding model for documents and queries, and treat the numbers as meaningful only relative to each other.
Explain it without notes
What is an embedding, and what problem does it solve?
Explain the difference between the dot product and cosine similarity. When do they give different rankings?
What is the difference between a model's internal token embeddings and an embedding model?
Name two mistakes that silently break semantic search.
Practice
Write cosine(a, b) for two lists and print cosine([1, 0], [0, 1]) and cosine([1, 2], [2, 4]) rounded to 4 decimals.
Write nearest(query, vectors) that returns the name with the highest cosine similarity to query. Use {"a": [1, 0], "b": [0, 1], "c": [1, 1]} and query [0.9, 0.2].
Trade-offs
- ↔
More dimensions capture finer meaning but cost more memory and search time.
- ↔
Embedding long documents as one vector is cheap but blurry, and embedding small chunks is precise but needs more vectors.
- ↔
A general embedding model is easy to start with, and a domain-specific one can rank better on your data.
Done when you can
Done when you can compute a dot product and a cosine similarity by hand.
Done when you can explain why cosine ignores length.
Done when you can explain why documents and queries need the same embedding model.
Done when you can name three uses of embeddings.