PHASE 3Intermediate ~23 min· topic 7 of 9

Topic 3.7

Filters Meet Vectors

In one line

Real searches say "only this user's documents" or "only the 2025 version". Filtering after an approximate search leaves too few results when the filter is selective, filtering before it can be slow or break a graph, and the practical answer depends on how many vectors pass the filter.

Think of it like this

You ask a librarian for the five books most like a book you love, but only from the shelf you own. If she first picks the five best books in the whole library and then crosses out the ones not on your shelf, you may be left with none. If she walks to your shelf and picks the best five there, you always get five, though she may have to look at every book on it.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Filter: A condition on a chunk's metadata that results must satisfy, such as owner = user 3.
Metadata: Facts stored with a chunk besides its text: owner, source, date, language, version.
Selectivity: The share of vectors that pass a filter. 5 percent means 1 in 20 pass.
Post-filter: Search first, then remove results that fail the filter.
Pre-filter: Restrict the search to vectors that pass the filter, then search.
Over-fetch: Asking the search for more than k results so that enough survive the filter.
Iterative scan: Continuing the index search in steps until enough results pass the filter.
Partial index: An index built only over rows that satisfy a condition, such as one tenant.

Step by step

01Step 1: give every document an owner

The example numbers 1,000 vectors and says document i belongs to user i % 20. The filter allowed(i, 20) is true for user 3's documents: 50 of them. A second run uses 200 users, so just 5 documents pass.

02Step 2: post-filter with a small fetch

Take the exact top 5 for a question, then keep only user 3's documents. Out of 5 results, about one in twenty belongs to user 3, so you are left with almost nothing: 0.12 results on average over 50 questions.

Step 2: post-filter with a small fetchdiagram

Rendering diagram…

03Step 3: over-fetch

Ask the search for 20, 100 or 500 results before filtering. The average climbs to 1.06, 4.24 and 5.00. But 100 is still not enough for every question (29 of 50 reach 5), and the cost of the search grows with the fetch size.

04Step 4: a stricter filter breaks it again

With 200 users only 5 documents pass the filter. Even fetching 500 gives 2.78 results on average and a full 5 for 5 of 50 questions. A filter that works in tests with generous data fails in production for the user with little data.

05Step 5: pre-filter and scan the allowed set

List the ids that pass the filter first (50 or 5 of them), then compute distances to just those and sort. Always 5 results (when at least 5 exist), 50 or 5 distance computations, and no wrong-user results.

prefilter.pywhole filepython

matching = [i for i in range(len(points)) if allowed(i, modulus)]
ranked = sorted(matching, key=lambda i: dist2(query, points[i]))
return ranked[:k]

06Step 6: pick the strategy by selectivity

A small function decides: few matching vectors, scan them exactly; many, use the index with an over-fetch sized from the selectivity. Real query planners (such as PostgreSQL's) make a similar choice from statistics. The practice at the end writes it.

Try it yourself

1
A generous filter
Change [20, 200] to [2, 20, 200]. With 2 users, half of the data passes. Which fetch size is enough now? Does the formula k / selectivity predict it?
2
Find the fetch size
For the 5-percent filter, add fetch factors until every one of the 50 questions gets 5 results. What is the smallest factor in your list that does it, and how many vectors would an index have to look at to produce that many?
3
Check for leaks
Read the last number on each "filter before the search" line. It should be 0. Why is it worth printing a number that should always be zero?

Code & diagrams

Filtering after and before the search New tab

The search itself is exact here, to keep the effect of the filter clear. With an approximate index the post-filter result can only be worse than shown.

Expected output

filter keeps 50 of 1000 documents
  filter after the search, fetch   5: 0.12 results on average, full 5 for 0 of 50 queries
  filter after the search, fetch  20: 1.06 results on average, full 5 for 0 of 50 queries
  filter after the search, fetch 100: 4.24 results on average, full 5 for 29 of 50 queries
  filter after the search, fetch 500: 5.00 results on average, full 5 for 50 of 50 queries
  filter before the search (50 distances): 5.00 results on average, 0 from the wrong user
filter keeps 5 of 1000 documents
  filter after the search, fetch   5: 0.04 results on average, full 5 for 0 of 50 queries
  filter after the search, fetch  20: 0.10 results on average, full 5 for 0 of 50 queries
  filter after the search, fetch 100: 0.62 results on average, full 5 for 0 of 50 queries
  filter after the search, fetch 500: 2.78 results on average, full 5 for 5 of 50 queries
  filter before the search (5 distances): 5.00 results on average, 0 from the wrong user

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget the filter

In the Python tab, change the body of allowed to return True.

terminal

$ Run

── what you'll see ──

filter keeps 1000 of 1000 documents

filter after the search, fetch 5: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 20: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 100: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 500: 5.00 results on average, full 5 for 50 of 50 queries

filter before the search (1000 distances): 5.00 results on average, 244 from the wrong user

filter keeps 1000 of 1000 documents

filter after the search, fetch 5: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 20: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 100: 5.00 results on average, full 5 for 50 of 50 queries

filter after the search, fetch 500: 5.00 results on average, full 5 for 50 of 50 queries

filter before the search (1000 distances): 5.00 results on average, 248 from the wrong user

Myth vs fact

Myth

Filtering a vector search is just a WHERE clause on top.

Fact

A WHERE clause applied after an approximate search can leave you with too few results, and applied before it can make the index slow or inaccurate. The strategy depends on selectivity.

Myth

If a test with the biggest user passes, the filter works.

Fact

The user with the least data is the one who gets empty results. Test with strict filters.

Myth

It is fine if the app drops forbidden results after the model answered.

Fact

The model has already seen the text and may quote it. Filter before the text reaches the prompt.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Measure the effective k: how many results survive after the filter, as a distribution over real queries and real tenants, and alert when many queries get fewer than k.
▸
Selectivity changes with time and tenant. A strategy chosen once is wrong later. The better systems estimate selectivity per query (from statistics or counts) and choose between an exact scan of the allowed set and the index.
▸
One index per tenant solves filtering for tenant isolation (and makes deleting a tenant trivial) but costs memory and operational overhead when there are thousands of tenants. A shared index with a tenant filter is the opposite trade.
▸
Combine filters with approximate-index parameters: a stricter filter often needs a larger efSearch or nprobe, and benchmarking (Topic 3.6) must be done with the filters production uses.

Remember this

1
Why this exists. Real RAG systems rarely search everything. A multi-user product searches only the signed-in user's documents. A help centre searches only the current product version, the reader's language, or articles published before today. An access rule must be applied *before* the model sees any text (Phase 10 comes back to this). Each of these is a filter on metadata: facts stored next to every chunk (Topic 4.6), such as owner, language, date and version.
2
Selectivity. The share of vectors that pass a filter is its selectivity. In the example 1,000 vectors belong to 20 users, so "only user 3" keeps 50 of them (5 percent), and a stricter filter with 200 users keeps just 5 (0.5 percent). Everything in this topic depends on that number.
3
Filtering after the search (post-filtering). Run the vector search for the top fetch results, then drop those that fail the filter. It is easy and works with any index. But if only 5 percent pass, a search for the top 5 leaves about 0.25 results. The example measures it: asking for 5 gives 0.12 results on average, fetching 20 gives 1.06, fetching 100 (that is 20 times k) gives 4.24 and a full 5 for only 29 of 50 questions, and fetching 500 finally gives 5 every time. A rule of thumb is to fetch about k / selectivity, plus a margin: 100 for 5 percent, 1,000 for 0.5 percent. For the 0.5 percent filter, even fetching 500 gives a full 5 for only 5 of 50 questions. pgvector's documentation describes the same effect: with approximate indexes the filter runs after the index scan, so "if a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average".
4
Filtering before the search (pre-filtering). Restrict the search to vectors that pass the filter, then take the top k among them. It always returns k results when enough vectors pass (5.00 in both cases in the example), and it never shows a wrong user's text. The catch is how to search only the allowed vectors with an index built over all of them. For a very selective filter the best answer is the simplest one: scan the allowed vectors exactly. Only 5 or 50 vectors need a distance computation, and an exact scan cannot miss.
5
Why indexes make pre-filtering hard. IVF can skip disallowed vectors inside the lists it opens, but a selective filter leaves those lists nearly empty, so you must open many more lists to find k allowed vectors. HNSW has a worse problem. If the walk is not allowed to touch disallowed nodes, the allowed nodes may no longer be connected to each other (the missing link of Topic 3.4's practice), and the walk dead-ends. If the walk may pass through disallowed nodes but only collect allowed ones, it must explore on the order of k / selectivity nodes to collect k results, so work grows as the filter gets stricter.
6
What real systems do. Four ideas appear again and again. One: choose by selectivity: scan exactly when few vectors pass, use the index with over-fetching when many pass (the practice below). Two: keep fetching until enough pass, called an iterative scan; pgvector 0.8.0 added hnsw.iterative_scan for this. Three: build a separate index per value when there are few distinct values (a partial index in pgvector, or a collection per tenant). Four: filter-aware traversal inside the graph or lists, offered by several vector databases. Topic 5.5 shows the filter syntax of real stores. Whichever you use, test it with a *strict* filter: the case that fails is the one with few matches.
7
Filters are also a security boundary. A forbidden passage must never reach the prompt. Apply the access filter inside the store's query, not by asking the model to ignore what it was shown and not by filtering in the app after the model has answered. The example prints how many returned results came from the wrong user: it must be zero.

Explain it without notes

Why does filtering after an approximate search return too few results, and how can you estimate how many to fetch?

What is pre-filtering, and why is it easy with a small allowed set but hard inside a graph index?

A team has one shared index for 5,000 tenants and tests only with their largest tenant. What goes wrong?

Why must access filters be applied before the prompt is built?

Practice

Write fetch_needed(k, matching, total) that returns how many results to fetch before filtering so that about k pass, rounded up (use integer arithmetic). Print it for k = 5, total = 1,000 and matching = 500, 50 and 5.

Write choose_strategy(k, matching, total): if matching is at most 100, return "scan the N allowed vectors exactly"; otherwise return "use the index, fetch F, then filter" where F is twice fetch_needed. Print the choice for matching = 5, 50 and 500 out of 1,000 with k = 5.

Trade-offs

↔
Post-filtering: simple and index-agnostic, but returns too few results when the filter is strict.
↔
Pre-filtering with an exact scan of the allowed set: exact and cheap for small sets, expensive when many vectors pass.
↔
One index per tenant: isolation and fast filtered search, at the cost of memory and operations for many tenants.
↔
Filter-aware indexes: the best of both, with more complex engines and more settings to benchmark.

Done when you can

Done when you can explain selectivity and compute the over-fetch needed for a post-filter.
Done when you can say why pre-filtering a graph index is hard.
Done when you can choose between an exact scan of the allowed set and an index search.
Done when you can write a test that checks the content of filtered results, not just their number.

Command Palette

Words you'll meet

Step by step

01Step 1: give every document an owner

02Step 2: post-filter with a small fetch

03Step 3: over-fetch

04Step 4: a stricter filter breaks it again

05Step 5: pre-filter and scan the allowed set

06Step 6: pick the strategy by selectivity

Try it yourself

A generous filter

Find the fetch size

Check for leaks

Code & diagrams

Break it on purpose

Forget the filter

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can