Topic 5.1
When a List in Memory Stops Being Enough
In one line
A list of vectors in a program searches fine until the program restarts, two people write at once, a question needs a filter, or there are millions of vectors. A vector database is the set of features that fix those, and you may not need one yet.
Think of it like this
A teacher keeps the class marks on one sheet of paper in her pocket. That works for one class and one teacher. Then the sheet is lost, two teachers want to write at once, the head wants "only Class 5B", and the school grows to a thousand classes. She needs a register, a cupboard, copies and rules about who writes when. A vector database is the register for embeddings.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Vector store / vector database
- A system that saves embeddings with ids and metadata, and finds the nearest ones to a query vector.
- Persistence
- Data survives a restart because it is written to disk.
- Upsert
- Insert a new item, or replace the existing item with the same id.
- Metadata (payload)
- Facts kept beside a vector, such as its document id, version or owner.
- Index
- An extra structure that makes search faster by skipping most vectors, usually at a small cost in accuracy.
- Multiply-add
- One step of a dot product: multiply two numbers and add the result to a running total.
- Exact search
- Compare the query with every vector. Always finds the true nearest ones.
Step by step
01Step 1: start with the list
The simplest store is a Python list of vectors and a loop that scores each against the query. It is the baseline for everything in this phase, and for Phase 3's exact search.
best = max(range(len(vectors)), key=lambda i: dot(query, vectors[i]))02Step 2: count the work instead of timing it
Wall-clock time depends on the machine. The number of multiply-adds does not. The example wraps dot with a counter: 1,000 vectors of 8 numbers cost exactly 8,000 multiply-adds. The cost is always vectors x dimensions.
03Step 3: scale the count to a real size
Small embedding models give vectors of a few hundred numbers (the lab model in this course gives 384). One million of them is 384 million multiply-adds per question. At an assumed one billion a second that is about 0.4 s on one core: too slow to feel instant, and the cost repeats for every user and every query.
04Step 4: list what else breaks
Restart: the vectors are gone. Edit a document: nothing updates the list. Two requests at once: unsafe. A condition such as "only version 2": you wrote a loop for it. Each is a feature to add, and together they are a database.
05Step 5: decide with numbers
Ask: how many vectors, how often do they change, do I need filters, can I rebuild in minutes after a crash? If the answers are "under a hundred thousand, rarely, no, yes", a list or a file is fine. Otherwise read on.
Try it yourself
- 1
Predict the count
Change the list size from 1000 to 2500 and the dimensions from 8 to 16. Before you run, write down the multiply-adds counted. Then run and check.
- 2
Change the assumed speed
In the table, replace the 1 billion with 10 billion (a library using SIMD may do more) and see which sizes stay under 100 ms.
- 3
Add a row
Add 100,000,000 to the list of sizes. In the JavaScript tab, is the count still exact? (Numbers are exact to about 9 quadrillion.)
Code & diagrams
The 1 billion multiply-adds per second is an assumption to make the table readable. Your CPU, the library (NumPy, SIMD) and the vector size change it a lot. The count itself is exact.
Expected output
searched 1000 vectors of 8 numbers
multiply-adds counted: 8000
best id: 464
same work at 384 numbers per vector (a common size for small models):
12 vectors -> 4608 multiply-adds, about 0.0 ms at 1 billion per second
10000 vectors -> 3840000 multiply-adds, about 3.8 ms at 1 billion per second
1000000 vectors -> 384000000 multiply-adds, about 384.0 ms at 1 billion per secondBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget that a search compares everything
In the Python tab, change max(range(len(vectors)), key=...) so the search stops at the first 100 vectors: max(range(100), key=lambda i: dot(query, vectors[i])). Run.
Myth vs fact
Myth
Every RAG app needs a vector database.
Fact
A few thousand vectors fit in a list or a file and search in microseconds to milliseconds. A database is for durability, updates, filters and scale, not for being able to say you have one.
Myth
A vector database makes answers more accurate.
Fact
It makes search faster and manageable. Accuracy comes from the embedding model, the chunks and the ranking. An approximate index can even lower accuracy slightly.
Myth
Brute force is always too slow.
Fact
Exact search over tens of thousands of vectors is fast on one core. The count tells you: vectors x dimensions.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
The first thing that breaks is rarely speed. It is operations: someone must rebuild vectors after a crash, and re-embedding everything costs money and time. Persisting vectors (a file, a table) is the cheapest first step.
- ▸
Keep the embedding model name and version beside the vectors. Vectors from two models, or two versions, are not comparable. A store without that field cannot tell you which vectors need re-embedding (Topic 5.4).
- ▸
If you already run PostgreSQL, adding vectors to a table (Topic 5.2) gives persistence, transactions and backups you already operate. The cost is that you now also tune that database for vector queries.
Remember this
- 1
Why this topic exists. Phase 3 kept vectors in a list and searched them by comparing the query with every one. That is correct and simple, and the right choice more often than people think. Before you add a new system to your product, you should be able to say which problem it solves. This topic lists those problems one by one.
- 2
Problem 1: persistence. A list lives in the program's memory. When the program restarts, redeploys or crashes, the vectors are gone and must be rebuilt, which means calling the embedding model for every passage again. A store writes vectors to disk so a restart only reloads them.
- 3
Problem 2: change while people search. Documents are edited and removed all day. A store needs upsert (insert, or replace if the id exists), delete, and rules for what a search sees while a write is half done. A plain list has none of this, and two threads changing it at once can corrupt it.
- 4
Problem 3: filters. Real questions come with conditions: only this user's documents, only the current version, only English. A store keeps metadata (also called payload) beside each vector and can apply a condition during the search (Topic 5.5).
- 5
Problem 4: scale. Comparing the query with every vector costs vectors times dimensions multiply-adds per search. The example counts it: for a million vectors of 384 numbers that is 384 million operations per question, about 0.4 seconds on one core at an assumed rate of a billion operations a second. An index (Phase 3) trades a little accuracy for a lot of speed, and a store builds and maintains it for you.
- 6
What a vector database adds, in one list. Durable storage, upsert and delete, metadata and filters, one or more indexes (HNSW, IVF), concurrent readers and writers, backups and replicas, and access control. When you do not need one: a few thousand vectors in a single process that can be rebuilt in seconds from source files is fine as a list or a file. Many products are in that state for a long time. Adding a database early costs operations work (accounts, upgrades, backups, monitoring) without making answers better.
Explain it without notes
List four problems that appear when you keep vectors only in a program's memory.
What does exact search cost, and how do you compute it?
What does a vector database add compared with a list?
When would you not add a vector database?
Practice
Write work(n, dims) that returns the multiply-adds of one exact search, and print it for 50,000 vectors of 768 numbers and for 2,000,000 vectors of 384 numbers.
Write fits_in_ms(n, dims, budget_ms, ops_per_second) that says whether one exact search fits in the time budget. Test it with 1,000,000 vectors of 384 numbers, a budget of 50 ms and 2 billion operations per second.
Trade-offs
- ↔
A list or file: nothing new to run and fully exact, but no concurrency, filters or scale.
- ↔
A database or extension: durability, filters and indexes, at the cost of operating it and keeping it in sync with your documents.
- ↔
An approximate index: much faster search, with a small chance of missing a true neighbour.
Done when you can
Done when you can name four problems a vector store solves.
Done when you can compute the cost of one exact search from the number of vectors and dimensions.
Done when you can say when a list or a file is enough.
Done when you can explain why vectors from different embedding models must not be mixed.