PHASE 2Beginner ~27 min· topic 9 of 10

Topic 2.9

Build: The Embedding Step

In one line

AskDocs version 2: an Embedder interface that the rest of the product depends on, a toy embedder behind it, vectors for the whole handbook, retrieval by cosine, and a cache keyed by content and model so nothing is embedded twice.

Think of it like this

A restaurant's prep kitchen. Before service starts, the cooks chop and portion every ingredient once and label the trays. During service, orders are quick because the slow work is done. Embedding the handbook ahead of time is the prep. Answering a question only needs one new tray, for the question itself.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Embedder: The part of the product that turns text into a vector. Everything else only calls its embed method.
Fake (test double): A simple stand-in with the same interface as the real thing, used so tests run fast and give the same result every time.
Cache: A store of results kept so the same work is not done twice.
Content hash: A short fingerprint of a text. The same text always gives the same hash, and a changed text gives a different one.
Cache key: What a cache entry is stored under. Here: model name plus content hash.
Re-embed: Compute the vectors again, for example after a model change.

Step by step

01Step 1: define the interface

Embedder has one method, embed(text), returning a unit vector. Python uses a base class, Java an interface, JavaScript a class with the same method. All AskDocs code depends on this and nothing else.

embedder.pywhole filepython

class Embedder:
    """Text in, a vector of length 1 out."""

    def embed(self, text):
        raise NotImplementedError

02Step 2: write the toy behind it

The table from Topic 2.1 grown to eight directions: account, refund, price, safety, time, repair, items, age. Embedding is: add up the direction of every known word, then scale to length 1. It counts its own calls so the example can show how much work was done.

03Step 3: embed the handbook once

At startup, run every passage through the embedder and keep the vectors in a dictionary by id. That is the prep. The example reports: 12 passages, 8 numbers each, 12 calls.

Step 3: embed the handbook oncediagram

Rendering diagram…

04Step 4: retrieve by meaning

Embed the question, score every passage with a dot product, keep those at or above MIN_SCORE, sort best first (ties keep handbook order) and cut to k. The shape is the same as version 0, so answer is unchanged.

05Step 5: read the trace

Run the five questions and read the sources and scores. Compare with version 1: the login question used to put refunds (D1) and the flat tyre (D10) above the password passage (D3), now D3 is first.

06Step 6: add the cache

Wrap any embedder in a CachedEmbedder: a key of model name:content hash, a list of the texts that are missing, one batched call for those, and the vectors returned in input order. The second program prints requests, texts sent and hits for four scenarios.

07Step 7: note what is next

Write down the open problems: a real model instead of the toy, chunking real documents, an index that does not scan every vector, exact words and hybrid search, and a measured threshold.

Try it yourself

1
Ask your own question
Add "Can I ride at night?" to the question list. Predict the sources and scores, then run. Which of the 8 directions does the toy not cover?
2
Raise the bar
Set MIN_SCORE to 0.75. Which questions lose which sources? Was every lost source one you would miss?
3
Teach the toy a word
Add "scooter" to the safety words and run the scooter question. What does it return, and is that a good answer? What does that tell you about a table written by hand?
4
Edit one text in the cache example
Change another text before the 'one text edited' run. How many requests and texts sent now? What does batch_size do to the request count?

Code & diagrams

AskDocs v2: embed the handbook, retrieve by meaning New tab

The embedder is a toy with a word table written for these questions. The structure (an Embedder interface, vectors built once, retrieve with the same shape) is what carries over to a real model.

Expected output

embedded 12 passages, 8 numbers each, 12 calls
Q: What does the monthly pass cost?
   sources: D8 (0.97), D12 (0.71), D6 (0.71)
   answer:  The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Q: I forgot my login, how do I get back in?
   sources: D3 (0.87), D5 (0.52)
   answer:  To reset your password, open the app, tap Forgot password and follow the link we email you. The link works for 30 minutes. [D3]
Q: Can I get my money back if I cancel?
   sources: D1 (0.97)
   answer:  A ride cancelled at least 2 hours before its start time is refunded in full. Later cancellations get ride credit instead. [D1]
Q: How long are lost things kept?
   sources: D9 (1.00)
   answer:  Items left on a bike are kept at the nearest shop for 14 days. [D9]
Q: Do you rent scooters?
   sources: none
   answer:  I can't find that in the handbook.
embedder calls in total: 17

A cache keyed by model and content, with batching New tab

FNV-1a is a small hash chosen so the three languages agree without libraries. Use SHA-256 in real code (hashlib in Python, MessageDigest in Java, the Web Crypto API or node:crypto in JavaScript).

Expected output

hash of 'Helmets': 222ddd49
first run         : 2 requests, 6 texts sent, 0 cache hits, 7 vectors returned
same texts again  : 0 requests, 0 texts sent, 7 cache hits, 7 vectors returned
one text edited   : 1 requests, 1 texts sent, 6 cache hits, 7 vectors returned
model switched    : 2 requests, 6 texts sent, 0 cache hits, 7 vectors returned

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget the model name in the cache key

In the Python tab of the cache example, change the body of key to return content_hash(text) (drop the model name). Run.

terminal

$ Run

── what you'll see ──

hash of 'Helmets': 222ddd49

first run : 2 requests, 6 texts sent, 0 cache hits, 7 vectors returned

same texts again : 0 requests, 0 texts sent, 7 cache hits, 7 vectors returned

one text edited : 1 requests, 1 texts sent, 6 cache hits, 7 vectors returned

model switched : 0 requests, 0 texts sent, 7 cache hits, 7 vectors returned

Myth vs fact

Myth

The embedding step is just one function call.

Fact

In a product it is an interface, a batch job, a cache, a record of the model used, and a check that the vectors are valid. That is why it gets its own phase.

Myth

Caching embeddings is a premature optimisation.

Fact

Re-embedding costs real money and time, and without a cache every deploy, every retry and every edit pays it again. A cache keyed by content is a few lines.

Myth

A cache key can be the text alone.

Fact

Then a model change reuses vectors from the old model and quietly corrupts search. The key must include the model.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Store the model name, version, dimensions and preparation (prefixes) with the index itself. When someone changes any of them, the index must be rebuilt, and the stored record is what lets a startup check refuse a mismatch (Topic 2.10).
▸
Use SHA-256 of the exact text (and anything that changes the embedding, such as a prefix) for the content hash. Normalise whitespace before hashing only if the embedder would also see the normalised text.
▸
Embed new or changed chunks incrementally and delete the vectors of removed ones. A full rebuild is the fallback when the model changes, and it should be a tested, one-command operation.

Remember this

1
What AskDocs can do after this phase, and what it could not before. Version 1 (Phase 1) matched words with BM25, so 'I forgot my login' ranked the refunds and flat-tyre passages above the password passage, because they share small words with the question. Version 2 matches meaning: the same question now finds the password passage first, with no shared word needed. It still has a toy embedder, no chunking, no index and no exact-word matching; the last section lists which phase adds each.
2
Why the product needs this now. This phase gave us meaning as vectors. Version 2 puts that into AskDocs without changing anything around it: retrieve(question, k) still returns scored hits, best first, and answer still builds a prompt, calls a model and cites sources.
3
What the step adds. An Embedder with one method, embed(text), that returns a unit vector. A ToyEmbedder behind it (the table from Topic 2.1, with eight meaning directions). The handbook embedded once when it loads. retrieve that embeds the question, scores every passage by dot product (which is cosine, since the vectors are unit length), and drops anything under MIN_SCORE = 0.5.
4
The interface is the point. The rest of AskDocs only knows Embedder.embed. Later you can swap in a real model, the browser model of the labs, or a hosted API, without touching retrieve, build_prompt or answer. Tests can use the toy as a fake: fast, free and the same every time. The example's second program is the other piece you will want in production, a cache.
5
The cache. Embedding costs time or money, and most texts do not change between runs. The CachedEmbedder stores each vector under model name + hash of the text. A text it has seen is never sent again, an edited text has a new hash and is embedded, and a new model name cannot see the old model's vectors. The run in the example: the first run sends 6 unique texts in 2 requests, the second sends nothing, one edited text costs 1 request, and switching the model re-embeds all 6.
6
Reading the trace. The example answers five questions. The pass-price question finds D8 at 0.97, but also D12 and D6 at 0.71, since they share price words ('fee', 'charged'): the toy's directions are coarse, and extra sources cost tokens. The login question finds D3 at 0.87 (and D5 at 0.52, for the shared 'app'). The money-back question finds D1 at 0.97. 'How long are lost things kept?' finds D9 at 1.00. The scooter question shares no word with the table, so nothing clears the bar and the answer is the refusal. The embedder is called 17 times: 12 for the handbook and 5 for the questions.
7
What is still missing. The toy table was written for these questions and would fail on others: a real model fixes that (see the lab in Topic 2.1). There is no chunking yet (Phase 4), the scan over all vectors is exact but slow for a big corpus (Phase 3 and 5), exact words still matter (Phase 6 adds hybrid search), and the threshold of 0.5 is a guess (Phase 8 and 9 set and measure it).

Explain it without notes

Why does AskDocs hide the embedding step behind an Embedder interface?

What does the cache key consist of, and why each part?

What changed in retrieve between version 0 and version 2, and what stayed the same?

In the trace, why does the pass-price question retrieve three sources? Is that a problem?

Practice

Write retrieve(question, k, min_score) over a dict VECTORS of id to unit vector, with a function embed(text) given. Return (id, score) pairs best first. Use the tiny two-direction table in the answer and print the result for the question 'password help'.

Write content_key(model_name, text) that returns model_name + ":" + sha256 hex of text and print the first 12 characters of the hash for 'Helmets' and for 'helmets'. What does it tell you about cache keys?

Trade-offs

↔
An interface from day one: a little design work now, easy swaps and fast tests later.
↔
A cache: saves cost and time on every rerun, at the price of a key that must include everything that changes the vector.
↔
Embedding at startup: simple, but slow for a large corpus. Production systems embed in a separate job and load the vectors (Phase 5 and 11).

Done when you can

Done when you can run AskDocs v2 in all three languages and explain each line of its trace.
Done when you can say what Embedder.embed takes and returns.
Done when you can explain what the cache key contains and why.
Done when you can list what version 2 still lacks and the phase that adds each.

Command Palette

Words you'll meet

Step by step

01Step 1: define the interface

02Step 2: write the toy behind it

03Step 3: embed the handbook once

04Step 4: retrieve by meaning

05Step 5: read the trace

06Step 6: add the cache

07Step 7: note what is next

Try it yourself

Ask your own question

Raise the bar

Teach the toy a word

Edit one text in the cache example

Code & diagrams

Break it on purpose

Forget the model name in the cache key

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can