Recipe: a baseline harness for any retriever
The retriever is a parameter. Pass a different function and the same questions measure it.
Two recipes to start any RAG project with: a decision function that says whether to paste, retrieve or fine-tune, and a baseline harness that measures how often your retriever finds the right passage. Plus where a real embedding call plugs in.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write hit_rate(results, expected, k) where results is a list of ranked id lists and expected a list of ids. Print the hit rate at 1 and at 2 for three made-up questions.
Extend recommend with a parameter access_controlled that adds the reason "documents have per-user permissions" and therefore recommends RAG. Print the decision for a 4-page document with permissions.
Explain it without notes
What does the baseline harness measure, and why is it written before improving retrieval?
What is the difference between hit rate at 1 and hit rate at k, and which matters more for RAG?
Why should recommend return reasons as well as a decision?
Expected output
place 1 expected D8, got D8, D10, D3 | What does the monthly pass cost?
place 1 expected D2, got D2 | When do shops close on public holidays?
place 1 expected D4, got D4, D1, D10 | Is a helmet included?
place 3 expected D3, got D1, D10, D3 | I forgot my login, how do I get back in?
place 2 expected D1, got D10, D1 | Can I get my money back if I cancel?
place 1 expected D10, got D10, D1, D4 | My tyre is flat, what now?
hit rate at 1: 4/6 = 0.67
hit rate at 3: 6/6 = 1.00