PHASE 5Intermediate ~20 min· topic 4 of 10

Topic 5.4

Collections, Ids and Payloads

In one line

A collection fixes the vector length and the distance when it is created. You choose the ids, and each point can carry a small payload. The full text usually stays in your own table, found by id.

Think of it like this

A filing cabinet drawer is made for one size of folder. You label every folder (the id), write a few facts on the tab (the payload), and keep the long papers in the main archive, not on the tab. A collection is the drawer: its size is fixed when you build it.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Collection: A named group of points that share a vector length and a distance (a table in SQL).
Point: One stored item: an id, a vector and a payload.
Dimension: How many numbers a vector has. Fixed per collection.
Payload: The metadata record stored beside a vector, such as title or version.
Stable id: An id that does not change when the same chunk is ingested again.
Model version: Which embedding model (and version) produced a vector. Store it.

Step by step

01Step 1: create the collection

Collection("lumen", 4, "cosine") fixes four numbers per vector and cosine distance. Nothing about the collection changes after this line. The toy embedder here gives four numbers (one per word group), a stand-in for a real model.

collection.pywhole filepython

lumen = Collection("lumen", 4, "cosine")

02Step 2: upsert with ids you choose

The handbook ids D1, D3 and D8 become point ids. Upserting D3 again replaces the point, so the count stays at three.

03Step 3: reject the wrong shape

A vector with three numbers fails with a clear message instead of being stored. Real stores do this too, and pgvector does it through the column type.

Step 3: reject the wrong shapediagram

Rendering diagram…

04Step 4: keep the text elsewhere

The payload holds the document id, the title and the model name. The text lives in the TEXTS table and is looked up by id after the search. The last two lines of the output show it.

05Step 5: write the schema down

As SQL it looks like this. The model column records which embedding model made the vector, so you can find vectors to re-embed later.

schema.sqlwhole filesql

CREATE TABLE chunks (
  id        text PRIMARY KEY,      -- stable: 'D3' or 'D3#2'
  doc_id    text NOT NULL,
  title     text NOT NULL,
  model     text NOT NULL,         -- which embedding model made the vector
  embedding vector(384) NOT NULL   -- dimension fixed here
);
-- the long text lives in documents(id, body), joined by doc_id

Try it yourself

1
Change the metric
Create the collection with the metric "dot". Do the scores change? Do the two results change?
2
Try a duplicate id
Upsert D8 with the vector and payload of the refunds passage. What does a search for the monthly pass return now, and what did the id mix-up lose?
3
Add a model check
Write a check in upsert that raises an error if payload["model"] differs from the collection's model field. Where would you store that field?

Code & diagrams

A collection: fixed shape, your ids, a payload, text elsewhere New tab

The 4-number embedder is a toy: one number per word group. A real model gives hundreds of numbers, but the collection behaves the same way.

Expected output

collection lumen: dim=4 metric=cosine points=3
after upserting D3 again: points=3
rejected: expected 4 numbers, got 3
D3 0.928 title=Password reset model=toy-4
   text: To reset your password, open the app, ta...
D1 0.000 title=Refunds model=toy-4
   text: A ride cancelled at least 2 hours before...

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Let an upsert with the wrong length through

In the Python tab, delete the two lines of upsert that check the length (the if len(vector) != self.dim: test and its raise). Run.

terminal

$ Run

── what you'll see ──

collection lumen: dim=4 metric=cosine points=3

after upserting D3 again: points=3

D3 0.928 title=Password reset model=toy-4

text: To reset your password, open the app, ta...

Traceback (most recent call last):

File "main.py", line 56, in <module>

print("%s %.3f title=%s model=%s" % (pid, s, payload["title"], payload["model"]))

~~~~~~~^^^^^^^^^

KeyError: 'title'

Myth vs fact

Myth

I can change the embedding model later and keep the same collection.

Fact

Different models produce vectors that are not comparable, even if the length matches. Build a new collection.

Myth

Let the database make up the ids.

Fact

Ids you cannot recompute make updates and deletes guesswork. Derive them from the document and chunk.

Myth

The text belongs in the vector store.

Fact

It can, but it does not have to. Keeping it in your main database avoids duplicate copies that drift apart.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Record the embedding model name and version in every payload (or in a table row). A migration then becomes a query: "all points where model is not the new one".
▸
Store only fields you filter on or show. Every payload field costs memory or disk, and some stores keep payloads in RAM.
▸
Distance and normalisation go together: if you normalise vectors to length 1, cosine and dot product give the same order and dot is cheaper. Decide once, document it, and enforce it at write time.

Remember this

1
Why this topic exists. Choices you make on the first day of a vector store, such as vector length, distance and ids, are painful to change later. Knowing what is fixed and what is free saves a rebuild.
2
What is fixed at creation. The dimension (how many numbers each vector has) and the distance (cosine, dot product, or L2). They must match your embedding model: a model that gives 384 numbers needs a collection of 384, and a collection cannot hold two models' vectors safely. In pgvector this is the column type vector(384). The example rejects a 3-number vector with expected 4 numbers, got 3.
3
Changing the model means a new collection. Vectors from two models, or two versions of one model, are not comparable even if they have the same length. The usual way to switch is to build a new collection, fill it, check it, and then move traffic to it (Topic 5.10).
4
Ids you control. Give every point an id that you choose and that stays stable across re-ingestion, such as docId#chunkNumber or a hash of the chunk's content (Topic 4.7). Then an upsert replaces the right point, a delete removes the right one, and a citation can point at the source. Letting the store pick random ids makes updates and deletes hard. The example upserts D3 twice and still has three points.
5
The payload. The payload (metadata) is a small record kept beside the vector: the source document id, title, version, language, owner or access label, and the embedding model name. Payload fields are what filters use (Topic 5.5). In some stores a field must be indexed to be filtered quickly.
6
Payload or external storage for the text? Storing the chunk text in the payload makes retrieval one call. Storing it in your own table keeps vectors small and the store light, and lets you edit text without touching vectors. A good rule: put in the payload what you filter on or must show without another lookup, and keep large or frequently edited text in your main database, found by id. Either way, the id is the link.
7
Unusual shapes. Some stores let a point have several named vectors (for example one for the title and one for the body), or sparse vectors for keyword-style search. Use them when you have a real reason; one vector per chunk is the normal case.

Explain it without notes

What is fixed when a collection is created, and why?

Why choose your own ids?

Where should the chunk text live, in the payload or in your own table?

How would you move to a new embedding model?

Practice

Write chunk_id(doc_id, n) that returns ids like D3#2, and parse_id(point_id) that returns (doc_id, n). Print both for D3, 2.

Write check_point(dim, vector, payload, model) that returns a list of problems: a wrong vector length and a payload whose model differs. Print the list for dim=4, a three-number vector and payload {"model": "old"} with model="new".

Trade-offs

↔
Text in the payload: one call to answer, bigger payloads and two copies to keep in sync.
↔
Text in your own table: one source of truth, one extra lookup.
↔
Cosine versus dot product: dot is cheaper and the same as cosine only when vectors are normalised.

Done when you can

Done when you can say what is fixed when a collection is created.
Done when you can design stable chunk ids.
Done when you can decide what goes into the payload and what stays in your database.
Done when you can describe how to migrate to a new embedding model.

Command Palette

Words you'll meet

Step by step

01Step 1: create the collection

02Step 2: upsert with ids you choose

03Step 3: reject the wrong shape

04Step 4: keep the text elsewhere

05Step 5: write the schema down

Try it yourself

Change the metric

Try a duplicate id

Add a model check

Code & diagrams

Break it on purpose

Let an upsert with the wrong length through

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can