Measure each stage on eight questions
Toy vectors and the TOY pair scorer, so only the method is real. The relation table of the scorer was written with these eight questions in mind: treat 8/8 as a demonstration, not a result.
Every extra stage costs code, time and money. Measure each stage's gain on your own questions with hit rate and reciprocal rank, and keep the simplest pipeline that is good enough.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write mrr(places) where places is a list of 1-based places (0 meaning missing) and print it for [1, 3, 0, 2] with 3 decimals.
Write compare(before, after) that takes two lists of places and returns the number of questions fixed (not first before, first after) and broken (first before, not first after). Print both for before = [1, 3, 2, 1] and after = [1, 1, 2, 2].
Explain it without notes
Define hit@1, hit@3 and MRR, and say what MRR shows that hit@1 does not.
In the example, hybrid had a lower hit@3 than keyword search. What does that teach?
When would you choose plain BM25 over hybrid search?
Why are the example's 8/8 results not evidence that re-ranking will work for you?
Expected output
keyword hit@1 4/8 hit@3 7/8 MRR 0.671
vector hit@1 2/8 hit@3 5/8 MRR 0.476
hybrid hit@1 6/8 hit@3 6/8 MRR 0.796
hybrid + re-rank hit@1 8/8 hit@3 8/8 MRR 1.000
re-ranking fixed: D4, D12
re-ranking broke: nothing