Topic 2.4
How Embedding Models Learn
In one line
An embedding model is trained on pairs: texts that belong together are pulled closer, texts that do not are pushed apart. 'Similar' therefore means 'similar for the task the model was trained on'. One vector per text is fast but cannot see how a question and a passage fit together.
Think of it like this
Seating guests at a wedding. You are told 'these two are friends, put them at the same table' and 'these two argued, keep them apart'. After many such instructions, the room arranges itself: friends close, rivals far. An embedding model arranges *texts* in space from a very large number of 'belongs together' and 'does not' hints.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Contrastive learning
- Training by comparing: pull matching pairs closer, push other pairs apart.
- Positive pair
- Two texts that should end up close, such as a question and its answer passage.
- Negative
- A text that should end up far from a given text.
- Margin
- The distance beyond which pushing apart stops, because the texts are already far enough.
- Hard negative
- A wrong text that looks very much like a right one. It is the most useful negative to train on.
- Bi-encoder
- A model that embeds each text separately. Fast, and the base of vector search.
- Cross-encoder
- A model that reads a question and a passage together and scores the pair. Accurate but slow, used to re-rank.
Step by step
01Step 1: start with arbitrary positions
Five words get random-looking 2D positions: login, password, refund, money, helmet. Nothing about their meaning is known yet. 'login' and 'password' are far apart (1.131) while 'login' and 'refund' are close (0.224).
02Step 2: state who belongs together
Two positive pairs: login with password, refund with money. Three negatives: login with refund, password with money, helmet with login. This is the *entire* supervision: no definitions, only 'close' and 'far'.
PAIRS = [
("login", "password", True),
("refund", "money", True),
("login", "refund", False),
("password", "money", False),
("helmet", "login", False),
]03Step 3: one training step
For each pair, look at the gap. If they belong together, move each point 10 percent of the gap towards the other. If they do not belong together and are closer than the margin (1.0), move each 10 percent away. Pairs already far apart are left alone.
04Step 4: repeat
After 10 steps the table has changed: login and password are 0.374 apart (from 1.131), refund and money 0.027 (from 0.781). The unrelated pairs are now 1.006, 0.618 and 1.127 apart. The structure came from the pairs alone.
05Step 5: see what training cannot give you
Only pairs we listed were arranged. A word with no pairs stays where it started. Real training uses so many pairs that nearly every sensible text has been placed, and the positions are in hundreds of dimensions, not two.
06Step 6: try a real model on a subtle case
In the lab, search for 'Do 15-year-olds have to wear a helmet?' against a passage that says riders under 16 must wear a helmet and one that says they must *not*. A bi-encoder scores both high, because it reads each passage alone and sees nearly the same words.
Try it yourself
- 1
Predict a distance
Before running, guess the distance between 'refund' and 'money' after 10 steps. Run. Then try 20 steps. What happens to a positive pair that has no push-apart pressure?
- 2
Add a hard negative
Add the pair
("refund", "password", False)to PAIRS. Both words already have positions. Does it change the 'login' and 'password' distance? Why or why not? - 3
Change the margin
Set
MARGIN = 0.3and run. Which negative pairs are pushed apart now, and which are left alone? What would a margin that is too small do to quality? - 4
Test a paraphrase in the lab
In the lab, how high does 'Children on bikes have to put on head protection.' score for the helmet question, compared with the 'must wear' passage? A paraphrase with no shared words still scores about 0.51.
Code & diagrams
A toy: five points, two dimensions, a simple update rule. Real models are neural networks with millions of parameters and use batches of thousands of pairs, but the idea (pull together, push apart) is the same.
Expected output
before training
login - password (belong together): 1.131
refund - money (belong together): 0.781
login - refund (unrelated): 0.224
password - money (unrelated): 0.141
helmet - login (unrelated): 0.566
after 10 steps
login - password (belong together): 0.374
refund - money (belong together): 0.027
login - refund (unrelated): 1.006
password - money (unrelated): 0.618
helmet - login (unrelated): 1.127Try: Press Search and compare the first two passages: one says riders under 16 must wear a helmet, the other that they must not. Note both scores. Then ask 'Do riders under 16 have to take their helmet off?': in our run the 'must not' passage wins by a hair (about 0.845 against 0.839). Why can a bi-encoder not reliably tell them apart?
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Pull every pair together, including the negatives
In the Python tab, replace the line sign = -1.0 # push apart, but only while they are closer than the margin with sign = 1.0 (pull the unrelated pairs together as well). Run.
Myth vs fact
Myth
Embedding models understand meaning the way people do.
Fact
They arrange texts from examples of what belongs together. That captures a lot of meaning but misses things that the examples did not teach, such as exact negation or precise numbers.
Myth
A model that is good on one benchmark is good on your data.
Fact
'Similar' is defined by the training pairs. A model trained on web questions may do badly on legal clauses or code. Test on your own questions (Topic 2.7).
Myth
More training pairs always means a better model.
Fact
Quality and variety of pairs, and especially hard negatives, matter more than raw count.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Real contrastive loss functions (such as InfoNCE) use all the other pairs in a batch as negatives at once, which is why large batch sizes help. The idea is the one in the example: raise the similarity of the true pair relative to every other.
- ▸
Because a bi-encoder embeds the passage without knowing the question, one vector has to serve every question that could ever be asked about it. A long passage on many subjects gets a blurred vector (Topic 2.6 and the chunking in Phase 4).
- ▸
Fine-tuning an embedding model on your own question and passage pairs is a real option when a general model underperforms on your domain. It needs good pairs, a held-out test set and a way to compare against the original (Phase 9).
Remember this
- 1
Why this exists. Topic 2.1 used a table written by hand. That cannot cover a language: nobody can write down the right numbers for every word and every phrase. Instead we let a model find the numbers from examples. The question is what the examples are and how the model uses them.
- 2
The training data: pairs. A positive pair is two texts that should be close: a question and the passage that answers it, a title and its article, two paraphrases. A negative is a text that should be far: a random other passage. Large models are trained on enormous numbers of such pairs, collected from the web, forums, question-and-answer sites and more.
- 3
The training loop. Start with random positions. For a positive pair, nudge the two positions towards each other. For a negative pair, if they are closer than a margin, nudge them apart. Repeat for a vast number of pairs. This is called contrastive learning. The example below does it by hand with five words in two dimensions: after 10 steps, 'login' and 'password' have moved from a distance of 1.131 to 0.374, and 'login' and 'refund' from 0.224 to 1.006.
- 4
Why negatives matter. Without a push-apart rule, the easiest way to make every positive pair close is to put everything at the same point. In the example, 'refund' and 'money' end up only 0.027 apart. Negatives stop the collapse by forcing unrelated texts away. Real training also uses clever choices of hard negatives: wrong passages that look right, which teach the finest differences.
- 5
'Similar' depends on the task. The model learns whatever similarity its pairs taught. If it was trained on question-to-passage pairs, 'close' means 'this passage answers this question'. If it was trained on paraphrase pairs, 'close' means 'says the same thing'. Those are different notions and a model good at one can be mediocre at the other. In the lab below, 'Riders under 16 must wear a helmet.' and 'Riders under 16 must not wear a helmet.' are almost as close as a pair of paraphrases (cosine about 0.74 and 0.73 against a helmet question), because topic and wording count more than a single 'not'.
- 6
One vector per text, and what it loses. The common design is a bi-encoder: the model reads the question alone and the passage alone, and each gets a vector. Passages can be embedded in advance, so search is fast. The price: the model never sees question and passage together, so it cannot reason about how they fit (negation, exact numbers, 'but' and 'unless'). A cross-encoder reads the pair together and is more accurate but too slow to run on a whole corpus. Phase 6 uses one to re-rank the shortlist.
Explain it without notes
What kind of data is used to train an embedding model, and what does the training do with it?
Why does contrastive training need negatives?
What does it mean that 'similar' depends on the training task?
What is the difference between a bi-encoder and a cross-encoder, and why does RAG use both?
Practice
Write a function pull(a, b, rate) that moves two 2D points towards each other by rate of their gap and returns the new points. Pull [0, 0] and [1, 0] with rate 0.25 and print both to two decimals.
Write push(a, b, rate, margin) that moves two 2D points apart by rate of the gap, but only if their distance is below margin. Print the result for [0, 0] and [0.5, 0] with rate 0.1 and margin 1, and for [0, 0] and [2, 0].
Trade-offs
- ↔
Bi-encoder: very fast search and embeddings computed ahead of time, but one vector cannot capture how a question and passage fit together.
- ↔
A general model: works across subjects out of the box, but may be weaker than a model fine-tuned on your domain.
- ↔
More negatives and hard negatives: a sharper model, at the cost of more complex and expensive training.
Done when you can
Done when you can explain contrastive training in two sentences.
Done when you can say why negatives are needed.
Done when you can explain what a bi-encoder loses compared with a cross-encoder.
Done when you have seen in the lab that a negation can score almost as high as the original.