Contrastive training by hand, in two dimensions
A toy: five points, two dimensions, a simple update rule. Real models are neural networks with millions of parameters and use batches of thousands of pairs, but the idea (pull together, push apart) is the same.
An embedding model is trained on pairs: texts that belong together are pulled closer, texts that do not are pushed apart. 'Similar' therefore means 'similar for the task the model was trained on'. One vector per text is fast but cannot see how a question and a passage fit together.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write a function pull(a, b, rate) that moves two 2D points towards each other by rate of their gap and returns the new points. Pull [0, 0] and [1, 0] with rate 0.25 and print both to two decimals.
Write push(a, b, rate, margin) that moves two 2D points apart by rate of the gap, but only if their distance is below margin. Print the result for [0, 0] and [0.5, 0] with rate 0.1 and margin 1, and for [0, 0] and [2, 0].
Explain it without notes
What kind of data is used to train an embedding model, and what does the training do with it?
Why does contrastive training need negatives?
What does it mean that 'similar' depends on the training task?
What is the difference between a bi-encoder and a cross-encoder, and why does RAG use both?
Expected output
before training
login - password (belong together): 1.131
refund - money (belong together): 0.781
login - refund (unrelated): 0.224
password - money (unrelated): 0.141
helmet - login (unrelated): 0.566
after 10 steps
login - password (belong together): 0.374
refund - money (belong together): 0.027
login - refund (unrelated): 1.006
password - money (unrelated): 0.618
helmet - login (unrelated): 1.127