Topic 2.1
From Word Counts to Meaning
In one line
A vector is just a list of numbers. The simplest vector for a text counts its words, one slot per word in the language, and it cannot tell that two different words mean the same thing. A dense embedding fixes that by giving every text a short list of numbers where nearby numbers mean nearby meaning.
Think of it like this
A map. To say where a cafe is, you do not list every street in the city and tick one. You give two numbers, a row and a column. Two cafes with nearby numbers are near each other. An embedding is a map reference for the *meaning* of a text: texts about the same thing get nearby numbers, even if they use different words.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Vector
- A list of numbers in a fixed order. An embedding is a vector.
- Bag of words
- A vector with one slot per word that exists. Each slot counts how often that word appears in the text.
- Sparse vector
- A vector where almost every number is 0, like a bag of words.
- Dense vector
- A short vector where every slot usually holds a non-zero number. Embeddings are dense.
- Dimensions
- How many slots a vector has. A bag of words has one per word in the language, an embedding has a few hundred.
- Embedding
- A dense vector that a model produces for a text so that similar meanings get similar vectors.
- Paraphrase
- The same meaning said in other words, such as 'forgot my login' and 'reset your password'.
Step by step
01Step 1: list every word
Take the question 'forgot my login' and the passage 'reset your password'. The words that exist across both are forgot, login, my, password, reset, your: six words, so six slots. Sorting them makes the order the same every time.
02Step 2: count into slots
The question has 1 in forgot, login and my. The passage has 1 in password, reset and your. Written as lists: question 1 1 1 0 0 0, passage 0 0 0 1 1 1. These are the bag-of-words vectors.
vocab = sorted(set(tokenize(QUESTION)) | set(tokenize(PASSAGE)))
def bag(text):
words = tokenize(text)
return [words.count(w) for w in vocab]03Step 3: compare, and see the flaw
Multiply slot by slot and add up (the dot product, Topic 2.2): every slot has a 0 on one side, so the total is 0. The score says 'nothing in common', although a person sees the same request. The problem is in the representation, not in the maths.
04Step 4: give every word a few shared dimensions
Make three slots called account, money and time. A table says what each word contributes: 'login' is [0.9, 0.1, 0.0], 'password' is [0.8, 0.0, 0.1]. The numbers were chosen by hand here. The point is that related words have similar lists.
05Step 5: a text is the sum of its words, scaled
Add up the lists of the known words in the text and scale the total to length 1 (Topic 2.3 explains why). The question 'forgot my login' becomes [0.959, 0.069, 0.274]: mostly account.
def embed(text):
total = [0.0] * len(DIMENSIONS)
for word in tokenize(text):
for i, x in enumerate(WORD_VECTORS.get(word, [0.0, 0.0, 0.0])):
total[i] += x
length = math.sqrt(dot(total, total))
return total if length == 0 else [x / length for x in total]06Step 6: compare again
The question now scores about 0.996 against 'reset your password' (both mostly account) and about 0.098 against 'get a refund' (money). The paraphrase is found, with no shared word. The knowledge sits in the table, and a real model has such a table for the whole language.
Try it yourself
- 1
Predict, then run
Before pressing Run, guess the similarity between 'forgot my login' and 'reset your password' in the dense part. Then run and compare. Which two slots do most of the work?
- 2
Make the table wrong
Change the vector of
passwordto[0.0, 0.9, 0.1](so it points at money). Run again. The similarity to the passage falls. You just saw that an embedding is only as good as the numbers behind it. - 3
Compare rankers in the lab
In the lab, note the cosine and the BM25 score of the 'lunch' passage. It shares the word 'forgot' with the question. Which ranker puts it above the password passage, and by how much?
- 4
Switch languages
Run the Java tab and compare it with Python. The bag-of-words lines are the same. Where does each language build the sorted list of words?
Code & diagrams
The dense table is a toy written by hand so you can read every slot. A real model learns hundreds of unnamed slots from data. The lab below uses a real one.
Expected output
vocabulary: forgot, login, my, password, reset, your
question: 1 1 1 0 0 0
passage: 0 0 0 1 1 1
shared words: 0
dimensions: account, money, time
question vector: [0.959, 0.069, 0.274]
similarity to 'reset your password': 0.996
similarity to 'get a refund': 0.098Try: Press Search. The password-reset passage should win although it shares only one word with the question. Then switch Rank by to Keyword (BM25) and see which passage keyword search puts first and why. Finally try the question 'Where can I get my account back?'.
Goals
- The password-reset passage is the best match
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Point 'password' the wrong way
In the Python tab, change the line "password": [0.8, 0.0, 0.1], to "password": [0.0, 0.9, 0.1], (so 'password' points at money). Run.
Myth vs fact
Myth
An embedding is a compressed copy of the text.
Fact
It is a position, not a copy. You cannot simply read the words off it, and two different texts can land very close together. It is still not safe to treat as anonymous: parts of the text can sometimes be reconstructed (Phase 10).
Myth
Each number in an embedding stands for one idea, like 'money' or 'time'.
Fact
Only in a toy. In a real model each slot mixes many things, and the meaning lives in how whole vectors compare.
Myth
A bag of words is useless.
Fact
It is the base of keyword search (Phase 1), it is fast, and it finds exact names and codes that embeddings blur. Phase 6 combines both.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
A bag of words over a real vocabulary has hundreds of thousands of slots, but you never store the zeros: sparse vectors are kept as (word, count) pairs. That is exactly an inverted index, the heart of Phase 1. A dense vector is the opposite trade: few slots, all stored.
- ▸
Dense embeddings are not 'better' in every case. They find paraphrases, but they can blur exact tokens such as product codes, error numbers and rare names. Systems that care about both use hybrid search (Phase 6).
- ▸
The vector length is a property of the model, fixed once and for all. Every vector in one index must come from the same model with the same length, or comparison is meaningless (Topic 2.10 turns this into a check).
Remember this
- 1
Why this exists. Search by words fails whenever a person says something in other words. 'I forgot my login' and 'reset your password' share no word at all, so a word-matching search scores them zero. We need to compare meaning, and a computer can only compare numbers. So we need a way to turn a text into numbers so that similar meaning gives similar numbers.
- 2
The simplest vector: a word count. Write down every word that exists in your texts. Give each word its own slot. A text becomes a list that holds, in each slot, how many times that word appears. This is a bag of words. It is a real vector, and it is sparse: nearly all of its slots are 0, because any one text uses only a few of all the words.
- 3
Its flaw: every word is a stranger. In a bag of words, 'login' and 'password' sit in different slots, and nothing says they are related. Two texts only look similar when they share the *same* words. The program has no idea that a word is close in meaning to another word. The example below shows it: the question and the passage have 0 shared words, so their score is 0.
- 4
The fix: dense vectors. Instead of one slot per word, use a small, fixed number of slots, say 384, and let *every* word and text use all of them. In the toy below, the slots are named (account, money, time) so you can read them. A word like 'login' gets a few large numbers in 'account' and small ones elsewhere. Words with related meaning get similar lists. Now 'forgot my login' and 'reset your password' point the same way and score about 0.996, while 'get a refund' scores about 0.098.
- 5
What is real and what is a toy. In a real model nobody names the slots and nobody writes the table. The model learns the numbers from huge amounts of text (Topic 2.4), and the slots mix many ideas at once. What stays the same is the contract: one text in, one list of the same length out, and closeness of lists means closeness of meaning.
- 6
What the numbers are not. A single number does not mean anything you could name, and a vector is not a summary or a translation of the text. You cannot read the words off it. It is a position. (Research shows that a determined attacker can sometimes rebuild much of a text from its vector, so treat vectors of private text as private data: Phase 10.) What matters is how two positions compare (Topic 2.2). LLM Engineering Topic 1.3 gives the same idea in one short topic; here we build it up step by step.
Explain it without notes
What is a bag-of-words vector, and why does it fail on 'forgot my login' versus 'reset your password'?
What is the difference between a sparse and a dense vector, and which one is an embedding?
Why do two texts with no shared words get a high similarity in an embedding space?
Can you recover the original text from its embedding? Why does it matter?
Practice
Write bag(text, vocab) that returns the word counts for a given vocabulary, and print the bag-of-words vectors for 'cancel a ride' and 'ride cancelled' over the vocabulary ["a", "cancel", "cancelled", "ride"]. How many words do they share, and what does that say about word counting?
Using the dense table {"cancel": [1, 0], "cancelled": [1, 0], "ride": [0, 1]}, write embed(text) that sums word vectors (unknown words add nothing) and scales to length 1, and print the embeddings of 'cancel a ride' and 'ride cancelled' to three decimals.
Trade-offs
- ↔
Bag of words: exact, fast, easy to explain and great for rare terms, but blind to meaning.
- ↔
Dense embeddings: find paraphrases and similar ideas, but are opaque, need a model and can blur exact tokens.
- ↔
Few dimensions: small and quick; many dimensions: more room for meaning, more memory (Topic 2.5).
Done when you can
Done when you can write a bag-of-words vector for a sentence by hand.
Done when you can explain why two paraphrases score 0 with word counts.
Done when you can explain what 'dense' and 'dimensions' mean.
Done when you can state what a vector's numbers do and do not mean.
Done when you have run the lab and read the cosine and BM25 columns.