PHASE 2Beginner ~22 min· topic 10 of 10

Topic 2.10

Playbook: Embeddings Done Right

In one line

Recipes to copy into any project: a batching embed client with retry and a unit-length check, a cache keyed by model and content, and a guard that refuses to query an index with the wrong model settings. Plus the checklist for changing models.

Think of it like this

A pilot's pre-flight checklist. Each item is small and boring. Together they stop the rare mistake that costs a lot. These recipes are the same: short, dull pieces of code that stop vectors from the wrong model, in the wrong shape, from reaching your users.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pre-flight check: A quick test that runs before the real work, to catch a known class of mistake.
Guard: Code that refuses to continue when a condition is not met.
Idempotent: Safe to repeat: doing it twice has the same effect as doing it once.
Jitter: A small random amount added to a wait, so many clients do not retry at the same moment.
Index settings: The model, dimensions, normalisation and prefixes stored with an index so later code can check it.

Step by step

01Step 1: batch and retry

Cut the texts into groups of at most the batch size. For each group call the API. If it says 'rate limited', wait (200 ms, then 400, then 800 ...) and try again, up to a maximum. The example's fake fails on its second call, so batch 2 needs two attempts and the client records a simulated wait of 200 ms.

retry.pywhole filepython

for attempt in range(1, max_attempts + 1):
    try:
        return api.embed(batch), attempt
    except RateLimited:
        if attempt == max_attempts:
            raise
        waits.append(base_wait_ms * 2 ** (attempt - 1))

02Step 2: normalise what comes back

Even if the provider says its vectors are unit length, scale them and check. The fake returns vectors that are not unit length on purpose, and the client prints that all 5 vectors have length 1 afterwards.

Step 2: normalise what comes backdiagram

Rendering diagram…

03Step 3: cache by model and content

Before calling the API, look up each text under model:hash. Send only the misses. See Topic 2.9.

04Step 4: store the settings with the index

Write down model name, dimensions, whether vectors are normalised, and the query and passage prefixes, next to the vectors they describe.

05Step 5: guard every start-up

Compare the stored settings with the live ones. If anything differs, block queries and say why. The example tries four situations: the same settings pass, a new model with a new size is blocked for two reasons, new prefixes are blocked, and a normalisation change is blocked.

06Step 6: plug in the real call

The fake API is replaced by a function that calls your provider (see the polyShow in Topic 2.8) and turns a 429 into RateLimited. Nothing else in the client changes.

Try it yourself

1
Make it fail harder
In the retry example, change the fake so it fails on the second and third call. How many attempts does batch 2 take, and what is the simulated wait? What happens at max_attempts of 2?
2
Add a length guard
Add a function too_long(text, limit_words) and call it before each batch. What should the client do with a chunk that is too long: cut it, skip it or stop? Defend your choice.
3
Extend the guard
Add a new field max_tokens to the index settings in the guard example. Add a scenario where it differs. Should a difference there block queries?

Code & diagrams

Recipe: batching embed client with retry, normalise and check New tab

Waits are simulated, not slept, so the example runs instantly. A real client sleeps for the wait, adds jitter and gives up after a maximum number of attempts.

Expected output

batch 1: 2 in batch, ok after 1 attempt(s)
batch 2: 2 in batch, ok after 2 attempt(s)
batch 3: 1 in batch, ok after 1 attempt(s)
5 vectors, all of length 1: yes
api calls: 4, simulated waiting: 200 ms

Recipe: refuse to query an index built with other settings New tab

Store the settings when you build the index and run this check at startup and before every rebuild.

Expected output

same settings: ok, safe to query
new model, new size: BLOCKED (2)
  - model changed (embed-v1 -> embed-v2): old and new vectors cannot be compared, re-embed everything
  - dimensions changed (384 -> 768): the index cannot hold these vectors
prefixes added: BLOCKED (1)
  - prefixes changed: queries and passages must be prepared the same way as when the index was built
vectors no longer normalised: BLOCKED (1)
  - normalisation changed (yes -> no): scores would no longer be cosine

Where the real call plugs inpython

Not run here: it needs a provider, a key and the network. Environment variables name the URL, the key and the model, so the code stays the same across providers that accept the OpenAI-style request.

import os
import httpx


class RateLimited(Exception):
    pass


def embed_api(batch):
    response = httpx.post(
        os.environ["EMBEDDINGS_URL"],
        headers={"Authorization": "Bearer " + os.environ["EMBEDDINGS_API_KEY"]},
        json={"model": os.environ["EMBEDDINGS_MODEL"], "input": batch},
        timeout=30,
    )
    if response.status_code == 429 or response.status_code >= 500:
        raise RateLimited(str(response.status_code))
    response.raise_for_status()
    rows = sorted(response.json()["data"], key=lambda row: row["index"])
    return [row["embedding"] for row in rows]

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Skip the normalisation

In the Python tab of the retry example, change vectors.extend(normalise(v) for v in raw) to vectors.extend(raw) (skip the normalising). Run.

terminal

$ Run

── what you'll see ──

batch 1: 2 in batch, ok after 1 attempt(s)

batch 2: 2 in batch, ok after 2 attempt(s)

batch 3: 1 in batch, ok after 1 attempt(s)

5 vectors, all of length 1: no

api calls: 4, simulated waiting: 200 ms

Myth vs fact

Myth

Checks like these are overkill for a small project.

Fact

They are a dozen lines each and catch the failures that produce wrong answers with no error. Small projects grow.

Myth

I can retry every failed call.

Fact

Retry rate limits, timeouts and 5xx errors. A wrong key or a too-long input fails the same way every time, and retrying only wastes time and money.

Myth

Upgrading to a newer version of the same model name is safe.

Fact

A new version can produce different vectors. Record the version beside the index and treat a change like a model change.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Put the index settings in the index itself or in a table next to it, not in code. Code changes with a deploy and the index does not, so the two drift apart unless the index remembers what it was built with.
▸
Add jitter to retry waits: with many workers, identical waits make them all retry at the same moment, which triggers the limit again (the thundering herd, LLM Engineering Phase 6).
▸
Report embedding job metrics: texts per second, tokens, retries, cache hit rate and failures. A drop in cache hit rate is often the first sign that a prefix or a normalisation setting silently changed.

Remember this

1
Recipe: the batching embed client. *When to use it:* any time you call an embedding API for more than one text. *What it guards against:* per-request overhead, rate limits and vectors that are not length 1. Split texts into batches below the provider's limits, retry a rate limit with growing waits, normalise every vector and check the result. The first example does it, with a fake that fails once.
2
Recipe: normalise and check. *When to use it:* where vectors enter your system. *What it guards against:* mixed lengths, which silently corrupt rankings (Topic 2.3). One function scales to length 1, another reports whether a vector is within 0.001 of length 1. Check on the way in and in tests. Re-normalise after anything lossy.
3
Recipe: the content cache. *When to use it:* always, once embedding costs anything. *What it guards against:* paying twice for the same text, and reusing vectors from another model. Key: model name plus a hash of the exact text. Embed only what is missing, once, in one batch, and return vectors in input order. Topic 2.9 has the full example. Adapt it by adding a persistent store (a file or database) so the cache survives restarts.
4
Recipe: the index guard. *When to use it:* at startup and before any index is built or queried. *What it guards against:* a model change, a dimension change, a prefix change or a normalisation change that nobody connected to the stored vectors. Store the settings next to the index and compare them with the live ones. The second example prints OK or BLOCKED with the reasons.
5
Recipe: the model-change checklist. *When to use it:* before switching or upgrading an embedding model. (1) Test the candidate on your question set and compare with the current model (Topic 2.7). (2) Check its prefixes, input limit, dimensions and licence. (3) Estimate the re-embedding cost with the token count and price. (4) Embed the whole corpus into a *new* index, with the settings stored beside it. (5) Run the test set against both and compare. (6) Switch queries and index together. (7) Keep the old index until the new one has served real traffic for a while.
6
Recipe: embed with a tokenizer-aware length check. *When to use it:* before every embedding job. *What it guards against:* silent truncation (Topic 2.6). Count tokens with the model's own tokenizer and log or fail on chunks over the limit. The first example shows the retry structure; add a too_long(chunk) check at the top of the loop.

Explain it without notes

What does a production embedding client do that a ten-line script does not?

Why store the index's settings next to the vectors?

Walk through the safe steps of switching to a new embedding model.

Why is checking that vectors have length 1 worth its own recipe?

Practice

Write with_retry(fn, attempts) that calls fn() and retries when it raises RuntimeError, up to attempts calls in total, and re-raises the last error. Test it with a function that fails twice and then returns "ok", and print the result and the number of calls.

Write check_settings(stored, live) that returns a list of the names of the keys whose values differ. Print the result for stored = {"model": "v1", "dims": 384} and live = {"model": "v2", "dims": 384}.

Trade-offs

↔
Strict guards (block on any difference): safe, but a harmless change can block a deploy until the check is updated.
↔
Retrying more times: higher success rate, but longer waits and more load on a service that is already struggling.
↔
A persistent cache: saves real money, but adds storage to manage and a key that must stay correct.

Done when you can

Before you ship: embeddings are requested in batches under the provider's limits, and rows are restored by index.
Before you ship: rate limits and server errors are retried with backoff and jitter, other client errors are not.
Before you ship: every vector is normalised and checked to be about length 1 where it enters the system.
Before you ship: embeddings are cached by model name and content hash, and only missing texts are sent.
Before you ship: the index stores its model, dimensions, normalisation and prefixes, and start-up refuses a mismatch.
Before you ship: chunks are counted in tokens and none is silently over the model's limit.

Command Palette

Words you'll meet

Step by step

01Step 1: batch and retry

02Step 2: normalise what comes back

03Step 3: cache by model and content

04Step 4: store the settings with the index

05Step 5: guard every start-up

06Step 6: plug in the real call

Try it yourself

Make it fail harder

Add a length guard

Extend the guard

Code & diagrams

Break it on purpose

Skip the normalisation

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can