Recipe: a recall benchmark harness
Candidates here are compressed searches on random vectors, but the harness accepts any function that returns ids and a work count: swap in IVF, HNSW or a database client.
Recipes to copy into any vector search project: test whether you need an index at all, start from documented parameters, benchmark against exact search, shrink and re-score, choose a filter strategy, and keep recall from silently dropping.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write check_recall_floor(recall, floor) that returns "ok: recall 0.970 meets the floor 0.950" when recall is at least the floor and "FAIL: recall 0.930 is below the floor 0.950" otherwise (3 decimals). Print the result for 0.93 and 0.97 with a floor of 0.95.
Write needs_index(n, dim, budget_ms) that estimates the exact query time as n * dim // 1000000 milliseconds and returns "yes" when twice that is above the budget, otherwise "no". Print the answer for 50,000, 200,000 and 1,000,000 vectors of 384 numbers with a 200 ms budget.
Explain it without notes
Walk through the test for whether a project needs a vector index.
Where would you start with IVF and HNSW parameters, and what do you do next?
What does the benchmark harness need, and why must it work with any search function?
What belongs in CI to protect recall, and what does it not protect against?
Expected output
exact recall@3 1.000 cheap 0 full 1000 meets target
2-bit codes only recall@3 0.617 cheap 1000 full 0 below target
2-bit + re-score 10 recall@3 0.917 cheap 1000 full 10 meets target
4-bit + re-score 10 recall@3 1.000 cheap 1000 full 10 meets target
4-bit + re-score 5 recall@3 0.950 cheap 1000 full 5 meets target
cheapest that meets the target (fewest full comparisons): 4-bit + re-score 5