Recipe: a retrieval test set and a threshold from data
Retrieval only, no model. Uses the fake embedder from Topic 8.3.
Five recipes to copy: a small service that chunks, indexes, retrieves, cites and refuses; a retrieval test set that measures hit rate and picks the refusal threshold from data; an incremental re-index; a prompt template; and the before-you-ship checklist.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write hit_at_k(results, expected, k) where results is a list of ranked name lists and expected is a list of names; return the number of questions whose expected name is in its first k results. Print it for k = 1 and k = 2 on three questions.
Write plan_sync(old, new) where both are dictionaries from name to fingerprint. Return three sorted lists: names to embed (added or changed), names to delete (only in old), and names unchanged. Print them for one example.
Explain it without notes
What two guards sit around the model call in a minimal RAG service, and what does each catch?
How would you choose the refusal threshold, and why not pick it by feel?
Why is ingestion made incremental, and what do you store to make it so?
What do hit@1 and hit@3 tell you that a handful of demo questions do not?
Expected output
top-1 miss: 'is shipping covered by the warranty' gave warranty, wanted shipping
hit@1: 10 of 11
hit@3: 11 of 11
threshold 0.20: answers 11 of 11 answerable, wrongly answers 1 of 4 unanswerable
threshold 0.50: answers 11 of 11 answerable, wrongly answers 1 of 4 unanswerable
threshold 0.80: answers 8 of 11 answerable, wrongly answers 0 of 4 unanswerable