PHASE 5Intermediate ~22 min· topic 5 of 10

Topic 5.5

Filtering in a Vector Store

In one line

A filter such as "only this user's documents" sounds simple, but an approximate index may collect its nearest candidates first and filter afterwards, leaving too few results. Stores fix it by filtering inside the search, by searching again with a wider net, or by splitting the data.

Think of it like this

You ask a librarian for "the five books closest to this idea, but only from the children's section". If she first collects the 20 closest books in the whole library and then throws away everything not in the children's section, you may get one book, or none. If she walks only the children's shelves, you get five. Filter-after and filter-first are those two librarians.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Filter: A condition on payload fields that every returned point must satisfy.
Post-filtering: Search first, then remove points that fail the filter. Can leave too few results.
Pre-filtering: Keep only points that pass the filter, then search among them.
Candidates: The nearest points an index collects before the filter removes some.
Iterative scan: Keep scanning the index for more candidates until enough pass the filter.
Partial index: An index that covers only the rows matching a condition.
Tenant: One customer or user whose data must be kept apart from the others.

Step by step

01Step 1: make a filter that keeps 5 percent

The example makes 200 random vectors and gives each an owner, user0 to user19, so each owner has 10 vectors. Asking for user7 keeps 1 vector in 20.

02Step 2: filter first

Keep user7's 10 vectors, rank them, take 5. Always 5 results, and only 10 comparisons. This is the reference answer the other methods are measured against.

03Step 3: filter after

Take the 20 nearest of all 200, then keep user7's. The line printed for 20 and for 40 candidates shows how many of the 5 wanted results were found. More candidates find more, at more cost.

Step 3: filter afterdiagram

Rendering diagram…

04Step 4: keep widening

Repeat the post-filter with a doubling number of candidates until 5 pass or everything has been looked at. The last two lines show how many candidates were needed and that the answer equals the filter-first answer. If the filter is strict, widening may need to look at nearly everything: then pre-filtering was the better plan.

05Step 5: write it in SQL

In pgvector the filter is an ordinary WHERE. Add a normal index for the filter field, and turn on an iterative scan or build a partial index if filtered queries come back short.

filtered.sqlwhole filesql

-- the filter is ordinary SQL
SELECT id, text
FROM chunks
WHERE owner = $2 AND version = 'current'
ORDER BY embedding <=> $1
LIMIT 5;

CREATE INDEX ON chunks (owner);   -- an ordinary index for the filter field

-- a partial index for a filter value that is in nearly every query
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
  WHERE version = 'current';

06Step 6: when the filter removes the best match

The lab below puts two versions of the refund rule next to each other. To the lab's embedding model they look almost the same. Against the question the 2025 passage scores about 0.52 and the 2023 passage about 0.51, and the two passages score about 0.99 against each other. The question cannot tell them apart; only a filter on a version field can.

Try it yourself

1
Make the filter stricter
Change i % 20 to i % 50 so that each owner has 4 vectors (2 percent), and run. Predict the change in the 20- and 40-candidate lines first. Then change K to 3 and see how the answers change.
2
Start the widening higher
Call keep_widening(who, 80). How many candidates does it end with? What is the cost of starting too high?
3
Another owner
Change who to user3. Is the filter-after result for 20 candidates better or worse? It depends on where that owner's vectors fall in the ranking: more evidence that post-filtering is unpredictable.
4
Use the lab
In the lab, reword the question to include the year, such as What is the 2025 refund rule? and search. Does the score of the 2025 passage rise enough to rely on? Why is a version field still safer than hoping the words help?

Code & diagrams

Filter after, filter first and keep widening New tab

The vectors are random numbers from a small fixed generator, so the example is the same in every language. 'Candidates' stands for what an approximate index collects before the filter.

Expected output

filter first (10 vectors compared): 107,147,127,187,87
filter after, 20 candidates: 2 of 5 found: 107,147
filter after, 40 candidates: 4 of 5 found: 107,147,127,187
keep widening from 20: 5 of 5 found after looking at 80 candidates
same answer as filter first: yes

Two versions of one rulereal embeddings

Try: Press Search. The 2023 and 2025 refund passages differ only in one number, so a real embedding model scores them almost the same against the question. Open the similarity grid and read how close the two passages are to each other. Then think: what must the store know to return only the current one?

QuestionPassages (4)

Rank byTop k (retrieved)

Embedding modelall-MiniLM-L6-v2384 numbers per text. Runs in your browser: the first Search downloads about 23 MB once, and your text never leaves this page.

Goals

The 2023 passage is in the top 2
The 2025 passage is in the top 2

Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the first filtered answer

In the Python tab, change the last two lines so they use filter_after(who, 20) as the final answer instead of keep_widening: replace got, size = keep_widening(who, 20) with got, size = filter_after(who, 20), 20. Run.

terminal

$ Run

── what you'll see ──

filter first (10 vectors compared): 107,147,127,187,87

filter after, 20 candidates: 2 of 5 found: 107,147

filter after, 40 candidates: 4 of 5 found: 107,147,127,187

keep widening from 20: 2 of 5 found after looking at 20 candidates

same answer as filter first: no

Myth vs fact

Myth

A filter just removes some results.

Fact

With an approximate index it can remove most of the candidates, leaving fewer than k. Where the filter runs matters.

Myth

Pre-filtering is always the right answer.

Fact

It is exact, but it can mean scanning a huge number of matching points. Stores mix the methods depending on how many points match.

Myth

The query text can pick the version for me.

Fact

Embeddings barely separate near-identical passages. Use a payload field and a filter.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Measure filtered recall, not just overall recall: run your recall benchmark (Topic 3.6) with the filters your product really uses, including a strict one.
▸
Apply the user's access filter on the server for every query. Do not rely on the prompt to hide documents: the model should never receive text the user may not see.
▸
Soft filters (a boost for recent documents) belong in ranking (Phase 6). Hard filters (security, tenant, version) belong in the store.

Remember this

1
Why this topic exists. Almost every real RAG question carries a condition: this user's files, the current version, this language, documents the asker may see. Topic 3.7 showed the idea on a small scale. Here is how real stores handle it, and the one way to get it quietly wrong.
2
Filter expressions. A filter is a condition on the payload: equality (owner = "user7"), membership (language IN ("en", "hi")), ranges (year >= 2024), and combinations with AND, OR and NOT. In SQL stores it is an ordinary WHERE. In dedicated stores it is a structured expression in the query. The meaning is the same everywhere: only points whose payload satisfies the condition may be returned.
3
Filter after (post-filtering). Ask the index for the candidates nearest points, then drop those that fail the filter. It is simple, and it breaks when the filter is strict. In the example, 5 percent of the vectors belong to user7. Twenty candidates contain about one of their vectors on average (two in this run), so asking for five returns fewer. Nothing fails; the user just gets thin results or, in a RAG product, an answer built on whatever was left.
4
Filter first (pre-filtering). Find the points that pass the filter, then rank only those. It is exact and always returns up to k results. With 10 matching vectors it compares 10 instead of 200. The catch is cost at scale: if a million points pass, you are doing brute force over a million, and a graph index such as HNSW cannot simply skip nodes without breaking its paths (Topic 3.7). Real stores combine the two with filter-aware traversals, bitmaps of matching points or separate indexes for common filters.
5
Keep widening. A pragmatic fix is to retry with more candidates until k results pass the filter, or until a limit. The example doubles the candidates (20, 40, 80, ...) and stops at 5 results. This is what pgvector's iterative index scans do for you (SET hnsw.iterative_scan = strict_order or relaxed_order; hnsw.max_scan_tuples caps how far it goes, default 20,000), according to the pgvector README. relaxed_order is faster but results may be slightly out of order.
6
Indexed filter fields. Filtering is fast only when the filter field is itself indexed: a normal B-tree index in SQL, a payload index in a dedicated store. An unindexed field means scanning every payload. Index the fields you filter on in almost every query, such as tenant, owner and version, and leave the rest.
7
Split the data instead. For a few very common filter values, put them in separate structures: a partial index (CREATE INDEX ... WHERE category_id = 123), a partitioned table, or one collection per tenant. Each query then touches only the part it needs. This is the cleanest answer for per-tenant search with many tenants of very different sizes, at the cost of more objects to manage.
8
Security note. Access control belongs in the filter, applied by your server from the signed-in user, never from text the user typed. A missing filter shows one user another user's documents. Test it with a user who owns nothing.

Explain it without notes

Why can a filter plus an approximate index return fewer than k results?

Compare pre-filtering and post-filtering.

What are three ways to fix short filtered results?

Why is the access filter applied on the server?

Practice

Write filter_after(items, query, candidates, wanted, k) where items is a list of (id, score, owner) already sorted best first. Return up to k ids from the first candidates items whose owner is wanted. Print the result for the list given in the answer.

Write widen(items, wanted, k, start) that doubles the number of candidates from start until k results are found or all items were looked at. Return (ids, candidates_used).

Trade-offs

↔
Post-filtering: fastest and simplest, unreliable for strict filters.
↔
Pre-filtering: exact and complete, can be slow when many points match.
↔
Partial indexes or partitions: fast and predictable for known filter values, more objects to maintain.

Done when you can

Done when you can explain why post-filtering returns short results.
Done when you can write filter-first, filter-after and widening from memory.
Done when you can name the pgvector settings for iterative scans.
Done when you can say why access control is a server-side filter.

Command Palette

Words you'll meet

Step by step

01Step 1: make a filter that keeps 5 percent

02Step 2: filter first

03Step 3: filter after

04Step 4: keep widening

05Step 5: write it in SQL

06Step 6: when the filter removes the best match

Try it yourself

Make the filter stricter

Start the widening higher

Another owner

Use the lab

Code & diagrams

Break it on purpose

Trust the first filtered answer

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can