A leak test that asks as every user
The guest has no groups on purpose. Unusual users are where filters break.
If some documents are for some people only, the check must happen inside retrieval, before the model sees anything. Every chunk carries who may read it, the search filters on that first, and a test asks as every kind of user.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write visible(chunks, groups) that returns the ids of chunks whose label is in groups, and returns an empty list when groups is empty. Test it with an empty set, {"public"} and {"public", "staff"}.
Write leaks(hits, groups) that returns the ids of hits whose label the user does not hold. Test it with the staff chunk and a public user.
Explain it without notes
Why is it not enough to tell the model in the system prompt not to reveal restricted documents?
Explain pre-filtering and post-filtering. What can go wrong with post-filtering?
What does "fail closed" mean for access control in retrieval, and what bug does it prevent?
How would you test, automatically, that a user never sees a chunk they are not allowed to see?
Expected output
buggy filter alice: 0 leaks, bob: 0 leaks, guest: 5 leaks
fail-closed filter alice: 0 leaks, bob: 0 leaks, guest: 0 leaks