Recall, precision and hit rate by hand
The rankings are written out as data so every number can be checked on paper. Averages divide by the number of questions, with no unanswerable ones in the set.
Three simple numbers say how well retrieval found the right passages in its top k. Each is computed here by hand, and the sweep over k shows which one matters for RAG and why.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write mean_recall(runs, k) where runs is a list of (ranked, relevant) pairs. Print it for two questions at k = 2.
Write smallest_k(runs, target) that returns the smallest k from 1 to 5 where mean hit rate reaches target, or None. Use the three runs in the answer.
Explain it without notes
Define recall@k, precision@k and hit rate@k and compute each for a question whose relevant set is {D11, D4} and whose top 3 is D4, D9, D11.
Why does precision usually fall as k grows, while recall can only rise?
Which of the three matters most for RAG and why? Is precision ever important?
How should unanswerable questions be handled in these averages?
Expected output
recall 1.000 precision 0.333 hit 1 | What does the monthly pass cost?
recall 1.000 precision 0.333 hit 1 | Can I get my money back if I cancel?
recall 1.000 precision 0.667 hit 1 | My daughter is 12, can she ride?
recall 0.000 precision 0.000 hit 0 | I forgot my login, how do I get back in?
recall 0.500 precision 0.333 hit 1 | My bike came back late and scratched
mean at k=3: recall 0.700 precision 0.333 hit rate 0.800
k recall precision hit rate
1 0.400 0.600 0.600
2 0.600 0.400 0.800
3 0.700 0.333 0.800
4 1.000 0.350 1.000
5 1.000 0.280 1.000