PHASE 8Intermediate ~23 min· topic 7 of 8

Topic 8.7

Vector Databases in Brief

In one line

A vector database stores embeddings and finds the nearest ones quickly. Exact search compares the query with every vector, which is fine for thousands of chunks; approximate indexes trade a little accuracy for speed at millions. Often a list in memory is enough.

Think of it like this

Finding a book in a huge library. With ten books you look at every cover. With ten million you do not walk every aisle: the library has sections, shelves and labels, so you go to the right section and scan only that. You might miss a book that was filed on the next shelf, but you find nearly everything in a fraction of the time. That is an approximate nearest-neighbour index.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Vector database: A system that stores embeddings and finds the nearest ones to a query vector.
Nearest neighbour search: Finding the stored vectors closest to a query vector.
Approximate nearest neighbour (ANN): Search that may miss some true neighbours in exchange for speed.
Recall: The share of the true nearest neighbours that a search actually returns.
Index: An extra data structure that makes lookups faster at the cost of memory and build time.
HNSW: A graph-based approximate index: it links vectors to near neighbours and walks the graph toward the query.
IVF: Clustering-based indexing: group vectors, then search only the closest groups.
pgvector: An extension that adds a vector type, distance operators and vector indexes to PostgreSQL.

Step by step

01Step 1: count the work

Exact search costs N × d multiplications per question. 5,000 chunks × 1,536 numbers is about 7.7 million: trivial. 50 million chunks is 77 billion: not trivial. The cost grows with the collection, which is what an index is for.

02Step 2: try the exact way first

The toy example begins with exact search over 500 random vectors and records the true top-3 for 20 queries. Every approximate result is judged against this answer.

03Step 3: group the vectors

Assign each vector to a bucket (its strongest dimension). For a query, find the buckets it leans toward and scan only those. Fewer comparisons, but a true neighbour in another bucket is never seen.

Step 3: group the vectorsdiagram

Rendering diagram…

04Step 4: turn the dial

Search 1, 2 and 3 buckets and compare the work (vectors scanned) with the recall (true neighbours found). More buckets means more work and better recall. Real databases expose the same dial under names such as probes or search width.

05Step 5: see the real SQL

With pgvector the whole thing is a table with a vector column, an index and an ORDER BY on a distance operator. The code block shows it in three languages. It is shown, not run, because it needs a PostgreSQL server.

Try it yourself

1
Read the dial
Run the example. For 1, 2 and 3 buckets, compute the share of the exact work done (scanned divided by 10,000) and the share of neighbours found (found divided by 60). Which setting would you pick for a chatbot? For a legal search?
2
Add a fourth setting
Add 5 to the list of bucket counts. Predict the recall before running. What is the recall with all 9 buckets, and why?
3
Sketch your own numbers
Take your own estimate: how many chunks will your documents make, and how many numbers per vector does your embedding model give? Multiply to see whether an in-memory list is enough.

Code & diagrams

A toy approximate index: work against recall New tab

500 random unit vectors from the course's tiny generator. Real indexes (HNSW, IVF) are far smarter, but the trade-off is the same.

Expected output

500 vectors, 9 dimensions, bucket sizes: 58 46 55 65 55 60 44 55 62
exact search: scanned 10000 vectors, found 60 of 60 true neighbours
search 1 bucket(s): scanned 1120 vectors, found 44 of 60 true neighbours
search 2 bucket(s): scanned 2259 vectors, found 59 of 60 true neighbours
search 3 bucket(s): scanned 3379 vectors, found 60 of 60 true neighbours

The same idea in PostgreSQL with pgvectorpython

Not run here: it needs PostgreSQL with the pgvector extension. The SQL (vector type, <=> cosine distance, HNSW index with vector_cosine_ops) is from the pgvector README; check it for the version you install.

# pip install "psycopg[binary]"      (needs PostgreSQL with the pgvector extension installed)
import os

import psycopg

# Three dimensions keep the example short. Use your embedding model's real dimension count (vector(1536) and so on).
with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
    conn.execute("CREATE TABLE IF NOT EXISTS chunks (id bigserial PRIMARY KEY, source text, body text, embedding vector(3))")
    # An approximate index (HNSW) with cosine distance. Without an index, search is exact but scans every row.
    conn.execute("CREATE INDEX IF NOT EXISTS chunks_embedding_idx ON chunks USING hnsw (embedding vector_cosine_ops)")

    conn.execute(
        "INSERT INTO chunks (source, body, embedding) VALUES (%s, %s, %s::vector)",
        ("returns", "Unused bikes can be returned within 30 days...", "[0.9,0.1,0.0]"),
    )

    query_vector = "[1.0,0.0,0.0]"  # the embedded question, written as text like [x,y,z]
    rows = conn.execute(
        "SELECT source, body, 1 - (embedding <=> %s::vector) AS similarity FROM chunks ORDER BY embedding <=> %s::vector LIMIT 3",
        (query_vector, query_vector),
    ).fetchall()
    for source, body, similarity in rows:
        print(source, round(similarity, 4))

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Search the wrong buckets

In the example, change the bucket choice to the weakest dimensions: replace (-q[d], d) with (q[d], d) in the sort that picks dims.

terminal

$ Run

── what you'll see ──

500 vectors, 9 dimensions, bucket sizes: 58 46 55 65 55 60 44 55 62

exact search: scanned 10000 vectors, found 60 of 60 true neighbours

search 1 bucket(s): scanned 1083 vectors, found 0 of 60 true neighbours

search 2 bucket(s): scanned 2203 vectors, found 0 of 60 true neighbours

search 3 bucket(s): scanned 3334 vectors, found 0 of 60 true neighbours

Myth vs fact

Myth

You need a vector database to do RAG.

Fact

Not at small scale. A few thousand chunks fit in a list in memory, and exact search over them takes milliseconds. Add a database when size, sharing, filtering or updates demand it.

Myth

Approximate search is inaccurate and therefore risky.

Fact

It is a measured trade: you can compute recall against exact search and tune it. High recall at a fraction of the cost is usually the right trade at scale.

Myth

The vector database makes retrieval good.

Fact

Retrieval quality comes mostly from chunking, the embedding model and query handling. A database only makes the search fast.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Memory is often the real limit: 1 million vectors of 1,536 float32 numbers is about 6.1 GB (1,000,000 × 1,536 × 4 bytes) before index overhead. Lower-dimension models, quantisation (smaller numbers) and disk-based indexes reduce it.
▸
In pgvector the distance operator in your query must match the index's operator class (<=> with vector_cosine_ops, <-> with vector_l2_ops, <#> with vector_ip_ops), and the ORDER BY must be the bare operator in ascending order. Otherwise PostgreSQL silently falls back to an exact scan: the answers stay correct but slow. Check with EXPLAIN. The recall dial is hnsw.ef_search for HNSW and ivfflat.probes for IVFFlat, and an IVFFlat index should be built after the table holds data, because it clusters what is there.
▸
Recall from Topic 8.3: for vectors of length 1, cosine, dot product and L2 distance give the same ranking, so pick the operator your index was built for. A plain index-less query is exact search, which is the right baseline for measuring recall.
▸
Approximate indexes are built from the data. Heavy insert or delete traffic can need re-indexing or lower the recall over time; check how your product handles updates.
▸
Pre-filtering versus post-filtering matters: filtering after a top-k search can leave you with fewer than k results. Test restrictive filters (one customer out of thousands).
▸
Keep the original text and metadata in a normal table keyed by id even if the vector store holds them too: you can rebuild the index with a new embedding model without losing anything.

Remember this

1
Why this exists. Exact vector search compares the query with every stored vector: with N chunks of d numbers, that is N × d multiplications for every question. For 5,000 chunks of 1,536 numbers it is about 7.7 million operations, milliseconds in memory. For 50 million chunks it is far too slow, and the vectors may not fit in memory. A vector database (or a database with a vector extension) stores vectors and answers nearest-neighbour queries efficiently, along with the things real systems need: persistence, updates, filters and access control.
2
Exact versus approximate. Exact (brute-force) search is always right and needs no index. Approximate nearest neighbour (ANN) indexes organise the vectors so a query examines only a small part of them. Two common families are clustering (group vectors, search the closest groups: IVF-style indexes) and graphs (link each vector to its neighbours and walk toward the query: HNSW). The price is recall: the share of the true nearest neighbours that the index actually returns. Settings such as the number of groups searched trade speed against recall.
3
The toy example shows the trade. 500 vectors are put in 9 buckets by their strongest dimension. Searching the single best bucket scans a fraction of the vectors and misses some of the true top-3; searching two or three buckets scans more and finds more. The exact numbers are printed below, and they come from a simple generator, so the same in all three languages. Real indexes are much cleverer than this, but the dial is the same: more work for better recall.
4
The categories. Three kinds of system store vectors. (1) A vector extension in a database you already run, such as pgvector for PostgreSQL: vectors sit beside your other tables, so you can filter with SQL and keep one system to operate. pgvector supports exact search and two approximate index types, HNSW and IVFFlat, and the distance operators <-> (L2), <#> (negative inner product) and <=> (cosine distance). (2) Dedicated vector databases built for this job, offered as managed services or open-source servers, with features such as scaling across machines and tuned indexes. (3) Libraries that run inside your process (an index in memory, saved to a file). Names change fast, so check each product's own documentation before choosing.
5
What else matters beyond speed. Metadata filtering (only chunks from this product or this customer), combined with vector search; updates and deletes without rebuilding everything; keyword plus vector (hybrid) search; access control per user; backups; cost at your size; and how the product behaves when the filter removes most of the candidates. Test with your real filters, because an approximate index plus a strict filter can return too few results.
6
When a list in memory is enough. If you have a few thousand chunks, load the vectors into an array and use the exact search from Topic 8.3. It is simple, exact, fast (milliseconds), easy to test and has nothing to operate. Reach for a database when the collection or the traffic outgrows memory, when many servers must share the data, or when you need filtering and updates at scale. The coming RAG course goes deeper on indexes, hybrid search and tuning recall.

Explain it without notes

Why not compare the query with every vector, and when is that fine?

What is recall in vector search and how do you measure it?

Give three reasons to prefer a vector extension in your existing database over a separate vector database, and one reason against.

Practice

A collection has 200,000 chunks and the embedding model returns 768 numbers (4 bytes each). How many megabytes of raw vector data is that (1 MB = 1,000,000 bytes)? Print it with one decimal.

Write recall(true_ids, found_ids) returning the share of true_ids that appear in found_ids. Print it as a percentage with one decimal for [1, 2, 3] versus [2, 3, 9].

Trade-offs

↔
Exact search is simple and correct but slows down with size; approximate search scales but needs tuning and monitoring of recall.
↔
A database extension keeps one system but may not scale as far as a dedicated store.
↔
More index settings (more probes, wider search) improve recall and cost latency and memory.

Done when you can

Done when you can estimate the cost and memory of exact search for your collection.
Done when you can explain exact versus approximate search and what recall means.
Done when you can name the categories of vector stores and one trade-off of each.
Done when you can decide whether an in-memory list is enough for a given project.

Command Palette

Words you'll meet

Step by step

01Step 1: count the work

02Step 2: try the exact way first

03Step 3: group the vectors

04Step 4: turn the dial

05Step 5: see the real SQL

Try it yourself

Read the dial

Add a fourth setting

Sketch your own numbers

Code & diagrams

Break it on purpose

Search the wrong buckets

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can