Topic 11.8
Changing the Embedding Model
In one line
Vectors from different embedding models live in different spaces and cannot be compared. Changing the model means building a second index, filling it in the background, measuring it on your test questions, switching only if it is better, and keeping a way back.
Think of it like this
A library re-labels every shelf with a new classification system. For a while, nobody can find anything if the catalogue uses the old codes and the shelves use the new ones. The sensible move is to build the new catalogue and label the shelves in a spare room, check that readers find books at least as easily, then open the new room. And keep the old one until you are sure.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Embedding space
- The set of all vectors one model can produce. Each model has its own, and they cannot be mixed.
- Backfill
- Re-processing all existing documents into a new index, in the background.
- Dual write
- Writing every change to both the old and the new index during a migration.
- Cut-over
- The moment the service starts using the new index for answers.
- Rollback
- Switching back to the old index after a cut-over that went wrong.
- Regression
- A question that worked before and does not work now.
- Canary
- Sending a small share of real traffic to the new version first to see how it behaves.
- Hit@1
- The share of test questions whose top result is the right passage.
Step by step
01Step 1: write down what the index was built with
Model name, version, dimension, chunker settings and the date. Store them with the index. If they are missing, a half-finished migration becomes a mystery.
02Step 2: show the danger
Embed the question with the new toy model and the documents with the old one. The scores are valid numbers and the ranking is wrong: D2, the opening hours, wins a question about the pass price with a cosine of 1.000.
03Step 3: add a guard
Compare the index's model tag with the query's model before searching and refuse on a mismatch. A loud failure in a test environment is far cheaper than a quiet one in production.
if index["model"] != query_model:
raise ValueError("index built with " + index["model"] + ", query embedded with " + query_model)04Step 4: backfill and dual write
Build the new index in batches while the old one serves. Anything edited meanwhile goes to both. The example prints progress per batch (3 of 8, 6 of 8, 8 of 8) and then shows an edit reaching both indexes.
05Step 5: measure, then decide with a written rule
Run the same test questions against both indexes. Cut over only if the new hit rate is not lower and no question regressed. Write the rule down before you look at the numbers, so you cannot bend it to fit.
06Step 6: cut over, watch, and keep the old index
Flip the pointer (or send a canary share first). Watch the metrics of Topic 11.6 for the new index: refusal rate, top score, latency. If anything looks wrong, flip back. Delete the old index only when you are sure.
Try it yourself
- 1
Break the guard
In the first example call
search(old_index, NEW, QUESTION)for the question "When do shops close on public holidays?". Does the mixed search still look wrong? Which questions hide the mismatch? - 2
Raise the bar
In the second example require the new index to have a hit@1 of at least 6 of 6. What blocks? Question 5 (the late bike) still fails: look at the words it shares with D6 and with D10 (hint: "bike" against "bikes"). Is that the new model's fault, or a limit of this word-matching toy?
- 3
Add a test question
Add
("Do helmets cost extra?", "D4")toTESTand run. Did the numbers change for the old index? Why does a bigger test set make a decision more trustworthy?
Code & diagrams
Two toy models with three numbers per word, one a reordering of the other. Real models differ in far more than order, and real scores are not these.
Expected output
old docs, old query: D2=0.000 D3=0.000 D8=0.894 -> top D8
new docs, new query: D2=0.000 D3=0.000 D8=0.894 -> top D8
old docs, NEW query: D2=1.000 D3=0.447 D8=0.447 -> top D2
guard: refused: index built with toy-v1, query embedded with toy-v2
guard: top D8Each candidate model is stood in for by a synonym table, so the measurement is real and reproducible. A real migration would embed with the new model instead.
Expected output
backfill new-1: 3/8
backfill new-1: 6/8
backfill new-1: 8/8
measure new-1: hit@1 4/6 (old 3/6), regressions: Q1
BLOCKED: still serving old
backfill new-2: 3/8
backfill new-2: 6/8
backfill new-2: 8/8
edit D8 written to: old, new-2
measure new-2: hit@1 5/6 (old 3/6), regressions: none
CUT OVER: serving new-2
ROLLBACK: serving old, old index still has 8 entriesBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Ignore regressions
In the Python tab of the second example, change if sum(new) >= sum(old) and not regressions: to if sum(new) >= sum(old): and run.
Myth vs fact
Myth
If the vector length matches, I can mix old and new vectors.
Fact
Length only says the arrays fit. The meaning of each position is different, so the similarity numbers are noise.
Myth
Re-embedding only the documents that changed is enough.
Fact
Changing the model changes the space for every document and every query. The whole corpus must be re-embedded.
Myth
A better benchmark score means a better system for us.
Fact
Public benchmarks measure other data and other questions. Only your own test questions, run on your own documents, tell you.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Some embedding models have tasks or prefixes (for instance, one for documents and one for queries, Topic 2.6). A migration must also copy those settings, or the new index silently uses the wrong mode.
- ▸
Matryoshka-style and truncated vectors let some models use shorter vectors with a small quality loss. If you shorten, record it in the index tag (Topic 2.5 covers dimensions) and shorten queries the same way.
- ▸
A shadow run sends real queries to the new index too, logs both results and does not use the new one. Comparing overlap of the top k (how many of the same ids appear) shows which queries change the most, before any user sees a difference.
- ▸
Budget for two indexes' memory and cost during the migration (Topic 11.5): both exist at once, and dual writes embed every change twice. Do the backfill sum too: the Topic 11.5 corpus is 6,000,000 tokens, which is 0.12 dollars at the example price, so money is rarely the limit. Time is. At an example limit of 1,000,000 tokens a minute the pass takes at least 6 minutes, and a corpus 1,000 times larger takes about 4 days, so batch size and the provider's rate limit decide the schedule.
- ▸
Changing the chunker, reranker or preprocessing follows the same recipe: a second index or version, a measured comparison, a pointer flip, a way back.
Remember this
- 1
Why you cannot just swap. An embedding model turns text into a vector (Phase 2). Each model defines its own space: the third number in one model's vector has no relation to the third number in another's. Even when two models produce vectors of the same length, comparing a vector from one with a vector from the other gives meaningless similarities. The first example shows it with toy models: the same question gets a perfectly fine answer under either model alone, and the wrong answer when its vector comes from the new model but the documents come from the old one. No error appears. The wrong document simply wins.
- 2
Why you change anyway. A better model may appear, a provider may retire the one you use, you may need another language, or cost may matter. Models also get new versions under the same name in some services, so check the provider's policy: a silent change is the same problem as a planned one, with no warning.
- 3
Rule 1: tag every index with the model that built it, and refuse mismatches. Store the model name, dimension and any settings (such as truncation) with the index. Before searching, compare it with the model that embedded the query. The guard in the first example refuses with a clear message instead of returning nonsense. Recording the tag costs nothing and is the best protection against a half-finished migration.
- 4
Rule 2: build the new index beside the old one. A backfill re-embeds every chunk with the new model into a second index, in batches, while the old index keeps serving. The cost is one full embedding pass (Topic 11.5) and the time it takes. While it runs, new and edited documents must be written to both indexes (a dual write), or the new index is stale the moment it finishes. The second example does the batches and the dual write.
- 5
Rule 3: measure before you switch. Use a fixed set of test questions with known right passages (Phase 9) and compare the two indexes: the overall hit rate (the share of test questions whose right passage is found), and the individual questions that got worse. A better average can hide a regression on questions that matter. The example blocks the first candidate (hit@1 4 of 6 against 3 of 6, but question 1 regressed) and accepts the second (5 of 6, no regressions). The rule it applies: cut over only if the total is not worse and nothing that worked before broke.
- 6
Rule 4: keep the way back. After the cut-over, keep the old index and keep dual-writing for a while. If something shows up in production that the test set missed (a new kind of question, a latency change), rolling back is flipping a pointer, not a rebuild. Delete the old index only after a period of calm. If the new model has a different dimension, also check memory (Topic 5.7): vectors twice as long need twice the room.
- 7
Cut over gradually if you can. Send a small share of traffic (say 5 percent) to the new index first, compare logs and user signals (Topic 9.8), then increase. This is the canary pattern. It also catches things offline tests miss, such as latency or odd inputs. The same pattern applies when you change the chunker (Phase 4), because new chunks also mean a new index.
Explain it without notes
Why can't you mix vectors from two embedding models?
Describe a safe migration to a new embedding model.
How would you decide whether to cut over, and what would stop you?
What is a dual write and why is it needed during a backfill?
Practice
Write check_same_model(index_model, query_model) that raises ValueError with the message index built with <a>, query embedded with <b> when they differ. Call it with two different names inside try/except and print the message.
Write overlap_at_k(old_ids, new_ids, k) returning the share of the top-k ids of the old list that also appear in the top-k of the new one, as a number, and print it with two decimals for ["D8", "D2", "D3"] against ["D8", "D3", "D4"] at k = 3.
Trade-offs
- ↔
Gradual cut-over: safer, and slower to finish and needs both indexes for longer.
- ↔
Strict regression rules: protect users, and can block a model that is better for most questions.
- ↔
Keeping the old index: fast rollback, at the price of memory and double writes.
Done when you can
Done when you can explain why vectors from two models cannot be compared.
Done when you can tag an index with its model and refuse a mismatch.
Done when you can describe a backfill with dual writes.
Done when you can state a cut-over rule that considers regressions, not only the average.
Done when you can describe how to roll back, and how long you keep the old index.