Command Palette

Search for a command to run...

Hectal
Phase 2Beginner3 of 14 in RAG and Vector Search

Embeddings in Depth

Turn text into vectors, compare them with cosine, store them, call embedding models safely, and build AskDocs' embedding step with a cache.

Phase 0 ended with a search that could not see that 'forgot my login' and 'reset your password' mean the same thing. This phase builds the fix from the ground up. You start with the simplest possible vector, a word count, see exactly why it fails, and then meet dense vectors, where nearby numbers mean nearby meaning. You compute dot product, cosine and distance by hand, learn why vectors are usually scaled to length 1, and watch a tiny model learn by pulling matching pairs together and pushing others apart.

Then the practical side: how much memory a million vectors take and how to shrink them, why a model silently ignores everything after its input limit, why some models want a different prefix for questions and for passages, how to choose a model by testing it on your own questions, and what a call to an embedding API looks like on the wire (request, response, batches, rate limits). LLM Engineering Topic 1.3 and Topic 8.3 introduce embeddings and vector search in one short topic each. This phase goes much deeper and does not repeat them.

You also run a real embedding model in your browser in four search labs, so every claim about paraphrases, negation and truncation is something you see, not something you are told. The phase ends with the Build topic, where AskDocs gets an Embedder interface, a fake for tests, a cache keyed by content, and meaning-based retrieval over the whole handbook, and with a playbook of recipes to copy.

0/10 · 0%
10 topics ~4 h 20 code blocks & diagrams
Start with the first topic
1
2.1

From Word Counts to Meaning

A vector is just a list of numbers. The simplest vector for a text counts its words, one slot per word in the language, and it cannot tell that two different words mean the same thing. A dense embedding fixes that by giving every text a short list of numbers where nearby numbers mean nearby meaning.

23 min 2 code practice

2
2.2

Dot Product, Cosine and Distance

Three ways to compare two vectors: multiply and add (dot product), compare the direction only (cosine), or measure the gap between the points (distance). They rank differently when vectors have different lengths and identically when every vector has length 1.

20 min 2 code practice

3
2.3

Normalising Vectors

Normalising a vector means scaling it to length 1 without changing its direction. Then the dot product equals cosine similarity, which is why most stores assume unit vectors. Mixing normalised and raw vectors silently corrupts a ranking.

19 min 1 code practice

4
2.4

How Embedding Models Learn

An embedding model is trained on pairs: texts that belong together are pulled closer, texts that do not are pushed apart. 'Similar' therefore means 'similar for the task the model was trained on'. One vector per text is fast but cannot see how a question and a passage fit together.

22 min 2 code practice

5
2.5

Dimensions and Storage

Every vector costs dimensions times bytes per number. A million 1536-dimension float32 vectors take about 6 GB. More dimensions can hold more meaning but cost memory and speed, and a few tricks (shorter vectors, smaller number types) cut the bill when the model allows.

20 min 1 code practice

6
2.6

Queries vs Documents

A question and the passage that answers it look very different: one is short and vague, the other long and full of detail. Some models want different prefixes for each, and every model reads only a limited number of tokens and silently ignores the rest.

31 min 2 code practice

7
2.7

Choosing an Embedding Model

Pick an embedding model by testing candidates on your own questions, not by reading a leaderboard alone. Weigh language, domain, input length, dimensions, cost, licence and where it runs. Changing the model later means re-embedding everything.

22 min 2 code practice

8
2.8

Calling Embedding Models

A hosted embedding model is one HTTP call: send a list of texts, get back a list of vectors plus a token count. Batch the texts, never trust the order of the rows, handle rate limits, and know the local alternatives (sentence-transformers, Transformers.js).

21 min 3 code practice

9
2.9

Build: The Embedding Step

AskDocs version 2: an Embedder interface that the rest of the product depends on, a toy embedder behind it, vectors for the whole handbook, retrieval by cosine, and a cache keyed by content and model so nothing is embedded twice.

27 min 2 code practice

10
2.10

Playbook: Embeddings Done Right

Recipes to copy into any project: a batching embed client with retry and a unit-length check, a cache keyed by model and content, and a guard that refuses to query an index with the wrong model settings. Plus the checklist for changing models.

22 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Length, normalise and check

Write length(v): the square root of the sum of the squares of the numbers in v.

Beginner
2

Rank vectors by cosine similarity

DOCS maps names to vectors (they do not have to be length 1).

Beginner
3

Plan the storage for a vector index

Write bytes_needed(n_vectors, dims, kind) for kind equal to "float32" (4 bytes per number), "int8" (1 byte per number) or "binary" (1 bit per number, so dims // 8 bytes). It returns a whole number of bytes. For any other kind raise ValueError.

Beginner
4

Embed in batches and restore the order

call_api(batch) stands in for an embeddings endpoint. It returns {"data": rows}, where each row is {"index": position in the batch, "embedding": [...]}, and the rows come back in reverse order.

Intermediate
5

Cache embeddings by content and model

embed_batch(texts) is a function you are given that returns one vector per text and costs money each call. Write the class CachedEmbedder(embed_batch, model_name) with a method embed_many(texts) that returns vectors in the order of texts.

Intermediate