Topic 2.7
Choosing an Embedding Model
In one line
Pick an embedding model by testing candidates on your own questions, not by reading a leaderboard alone. Weigh language, domain, input length, dimensions, cost, licence and where it runs. Changing the model later means re-embedding everything.
Think of it like this
Hiring a translator. Résumés and test scores help you shortlist, but you give your top two candidates a page from *your* documents and see who does it well. A public benchmark is the résumé. Your own questions are the page.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Hosted model
- A model you call over the network from a provider, and pay for by the token.
- Open weights
- A model whose files you can download and run yourself. The licence says what you may do with it.
- Benchmark / leaderboard
- A public test that scores many models on shared datasets. Useful for shortlisting.
- MTEB
- A public collection of embedding tests and a leaderboard of models on Hugging Face (the Massive Text Embedding Benchmark).
- Hit rate at k
- How often the right passage appears in the top k results (Topic 0.8).
- Re-embedding
- Running every stored text through a model again, after a model change.
Step by step
01Step 1: list what you need
Write down the language(s), the type of text, the typical length of a chunk, the corpus size, the budget, and the privacy rule (may the text leave your system?). These cut the list of models before you test any.
02Step 2: shortlist from a leaderboard
Use a public leaderboard as a map and take three or four candidates that fit your list: for example one small and cheap, one strong and hosted, one open and mid-sized. Check each model's card for prefixes, input limit, dimensions and licence.
03Step 3: write your own test set
Collect real questions, each paired with the id of the passage that answers it. The example uses four. A real set has dozens, taken from support tickets, logs or interviews.
TESTS = [
("Can I get my money back if I cancel?", "D1"),
("I forgot my login", "D3"),
("Is a helmet included?", "D4"),
("What does the monthly pass cost?", "D8"),
]04Step 4: measure each candidate the same way
Embed the same passages with the same chunking and measure the place of the right passage for each question. The small toy model reaches the right passage first for 2 of 4 questions, the large one for 4 of 4.
05Step 5: put a price on it
Cost = tokens embedded divided by a million, times the price per million tokens. The example uses example prices (0.02 and 0.13 dollars). Include re-embedding after each document update and a possible future model change.
06Step 6: decide, and write down why
Record the model name, version, dimensions, prefixes, limit and the test results next to the index. In a year, someone will ask why this model was chosen, and the notes make a change safe.
Try it yourself
- 1
Add a harder question
Add
("Do I pay if the bike gets damaged?", "D5")to the tests and add the D5 passage to the documents. Does either toy model get it right? What would you have to add to the large table? - 2
Change the price
Set the large model's price to 0.02 and the corpus to 2,000,000 chunks. How does the cost compare? At what price difference would you accept the smaller model?
- 3
Use the lab as a test
In the lab, replace the question with five different real-sounding questions about one passage. Count how many keep it first. That is a miniature hit rate.
Code & diagrams
Both models are toys with word tables, and the prices are example values (check your provider). The method, not the numbers, is what carries over.
Expected output
small model: places [3, 4, 1, 1], first place for 2 of 4
large model: places [1, 1, 1, 1], first place for 4 of 4
embedding 50000 chunks with the small model: $0.40
embedding 50000 chunks with the large model: $2.60Try: Search for the refund question and note the score of the refunds passage. Then try 'Can I get a refund?', 'What is your cancellation policy?', 'cash back for cancelling a booking' and 'my money'. Which wordings keep the right passage first? Which fails, and why?
Goals
- The refunds passage is first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Test with a single easy question
In the Python tab, delete the first three lines of TESTS so only the monthly pass question is left. Run.
Myth vs fact
Myth
The model at the top of the leaderboard is the best for me.
Fact
A leaderboard average hides the tasks and the language you care about. Use it to shortlist, then test on your own questions.
Myth
A bigger or more expensive embedding model is always better.
Fact
Often, but not always, and the gain on your data may be tiny. Measure. Cost grows with tokens, dimensions and storage.
Myth
I can swap models later without work.
Fact
Vectors from different models are incompatible, so a swap means re-embedding everything and rebuilding the index.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure on retrieval metrics (hit rate, MRR, nDCG: Phase 9), with fixed chunking, and include at least a few hard cases such as paraphrases, numbers and negations. A model that wins on easy questions can lose on the rest.
- ▸
Check the licence of open-weights models. 'Open weights' does not mean 'free for any use': some licences limit commercial use or require attribution. The lab's model is under the Apache-2.0 licence.
- ▸
Plan the migration before you need it: keep the raw text, store the model name and dimension with every vector, build the new index beside the old one, compare on the test set, and switch traffic only when the new index wins.
Remember this
- 1
Why this exists. There are many embedding models, hosted and open, small and large, English-only and multilingual. They are not interchangeable: vectors from different models cannot be compared with each other, and a model that is best on a public test may do poorly on your text. The choice also sticks, because everything you index is tied to it.
- 2
Hosted or open weights. A hosted model is called over the network: no hardware to run, strong models, a price per million tokens, rate limits, and your text leaves your system. An open-weights model is downloaded and run by you: no per-call fee, full control and privacy, but you run the hardware and pick a size that fits it. The model in this course's lab is an open-weights model small enough to run in a browser.
- 3
What to compare. Language support (English only, or many languages), domain (general web text, or code, legal, medical), the maximum input length (Topic 2.6), the number of dimensions (Topic 2.5), whether it needs prefixes, its licence (can you use it commercially?), speed, and cost. Prices and sizes change often: look them up when you decide, and never copy them from an old article.
- 4
Leaderboards are a map, not a verdict. Public benchmarks, such as the MTEB leaderboard on Hugging Face, rank models on many datasets. They tell you which models are worth testing. They do not tell you which is best for a bike-rental handbook. Models can also be tuned to the public tests, and averages hide the tasks you care about.
- 5
Test on your own questions. Write 30 to 100 real questions with the passage that answers each (Topic 0.8 started this). Embed your corpus with each candidate, and measure how often the right passage is first and in the top few. The example does this with two toy models: the small one puts the right passage first for 2 of 4 questions (places 3, 4, 1, 1) and the large one for 4 of 4. Write down cost beside it: at example prices of 0.02 and 0.13 dollars per million tokens, embedding 50,000 chunks of 400 tokens costs about 0.40 and 2.60 dollars.
- 6
Changing the model later. Because vectors from different models are incompatible, switching means re-embedding the whole corpus and rebuilding the index, often while the old one keeps serving. It is the reason to choose with care, keep the raw text, and record the model name beside every vector (Topic 2.9 and 2.10).
Explain it without notes
How would you choose an embedding model for a new RAG project?
Why are public leaderboards not enough to choose a model?
What is the cost of switching embedding models after launch?
What are the main differences between hosted and open-weights embedding models?
Practice
Write first_place_rate(places) that takes a list of places (1 is first) and returns the fraction that are 1. Print it for [3, 4, 1, 1] and [1, 1, 1, 1] as a percentage with no decimals.
Write embedding_cost(chunks, tokens_per_chunk, price_per_mtok) and print the cost to embed 120,000 chunks of 350 tokens at an example price of 0.05 dollars per million tokens, to two decimals.
Trade-offs
- ↔
Hosted models: strong and easy to start with, but cost per token, rate limits and data leaving your system.
- ↔
Open-weights models: private and no per-call fee, but you run them and the best ones need real hardware.
- ↔
Bigger models: usually better recall, higher cost in money, memory and latency.
Done when you can
Done when you can list the factors that decide a model choice.
Done when you have a written test set of real questions with their right passages.
Done when you can compute the cost of embedding a corpus at a given price.
Done when you know what switching models later would involve.