Topic 9.3
Recall@k, Precision@k and Hit Rate
In one line
Three simple numbers say how well retrieval found the right passages in its top k. Each is computed here by hand, and the sweep over k shows which one matters for RAG and why.
Think of it like this
You ask a librarian for books about bikes. She brings back three. Of the five good books in the library, two are in her pile. Recall asks how many of the good books she found (2 of 5). Precision asks how much of her pile was good (2 of 3). A librarian who brings the whole library has perfect recall and terrible precision; one who brings a single sure book has perfect precision and poor recall.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Relevant passage
- A passage the golden set says helps answer the question.
- Top k
- The first k passages of the ranked results, the ones the model will read.
- Recall@k
- Relevant passages found in the top k, divided by all relevant passages.
- Precision@k
- Relevant passages found in the top k, divided by k.
- Hit rate@k
- The share of questions with at least one relevant passage in the top k.
- Mean (average) over questions
- Compute the metric for each question, then average the numbers. Every question counts equally.
Step by step
01Step 1: count how many relevant passages are in the top k
All three metrics start from one count. Slice the ranked list to k and count the ids that are in the relevant set.
def found_in_top(ranked, relevant, k):
return sum(1 for doc_id in ranked[:k] if doc_id in relevant)02Step 2: divide three different ways
Recall divides by the number of relevant passages, precision by k, hit rate asks only whether the count is above zero. Three one-line functions.
def recall_at_k(ranked, relevant, k):
return found_in_top(ranked, relevant, k) / len(relevant)
def precision_at_k(ranked, relevant, k):
return found_in_top(ranked, relevant, k) / k
def hit_at_k(ranked, relevant, k):
return 1 if found_in_top(ranked, relevant, k) > 0 else 003Step 3: score each question, then average
The first block of the example prints one line per question. The login question is the miss: D3 sits at rank 4, so all three numbers are zero at k = 3.
04Step 4: read the multi-passage question
'My daughter is 12' needs D11 and D4. With both in the top 3, recall is 1.000 but precision is 0.667 because D9 came along. The late-and-scratched question needs D6 and D5, and only D6 is in the top 3: recall 0.500, hit 1. Hit rate calls that a success; recall shows it found half of what was needed.
05Step 5: sweep k
Run the same metrics for k = 1 to 5. Recall and hit rate can only go up as k grows. Precision generally falls. The k where recall stops improving (here 4) is a sensible choice, balanced against the token cost of each extra passage.
06Step 6: compare systems at the same k
A metric is only meaningful against another run on the same golden set and the same k. Phase 9's later topics turn that comparison into a gate (Topic 9.7).
Try it yourself
- 1
Compute one by hand first
For the late-and-scratched question the ranking is D6, D3, D2, D5, D1 and the relevant set is D6, D5. On paper, write recall, precision and hit at k = 2, 3 and 4. Then check against the code.
- 2
Fix the miss
Move D3 to rank 3 in the login question's list. Which three means change at k = 3, and by how much?
- 3
Add an unanswerable question
Add
("Do you rent scooters?", ["D1"], set())style data with an empty relevant set and run. What happens inrecall_at_k? Decide how your evaluation should treat such a row. - 4
Use the lab
The lab starts with the refund passage written as vague office language. Rewrite the text after 'Policy 4.2:' (keep the label) so that the question finds it in the top 3 and read the new rank. What did you have to put into the text?
Code & diagrams
The rankings are written out as data so every number can be checked on paper. Averages divide by the number of questions, with no unanswerable ones in the set.
Expected output
recall 1.000 precision 0.333 hit 1 | What does the monthly pass cost?
recall 1.000 precision 0.333 hit 1 | Can I get my money back if I cancel?
recall 1.000 precision 0.667 hit 1 | My daughter is 12, can she ride?
recall 0.000 precision 0.000 hit 0 | I forgot my login, how do I get back in?
recall 0.500 precision 0.333 hit 1 | My bike came back late and scratched
mean at k=3: recall 0.700 precision 0.333 hit rate 0.800
k recall precision hit rate
1 0.400 0.600 0.600
2 0.600 0.400 0.800
3 0.700 0.333 0.800
4 1.000 0.350 1.000
5 1.000 0.280 1.000Try: Think of this as one row of a golden set: the question should find the refund passage in its top 3. Press Search. The refund passage is written in office language, so it ranks low. Rewrite what comes after the label 'Policy 4.2:' (keep the label, because the goal finds the passage by it) in the plain words a rider would use: the rule is cancelling at least 2 hours before the start for a full refund. Search again and watch its rank and its cosine score. A rewrite such as 'Policy 4.2: if you cancel at least 2 hours before the ride starts, you get a full refund.' scores about 0.54 against this question and moves from result 6 to result 1. What did you put in the text that the question was waiting for?
Goals
- The refund passage is in the top 3
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Divide recall by k
In the Python tab, change the last line of recall_at_k to divide by k instead of len(relevant): return found_in_top(ranked, relevant, k) / k. Run.
Myth vs fact
Myth
Precision should be as close to 100 percent as possible.
Fact
With one relevant passage and k = 3 the ceiling is 33 percent. Compare precision between systems at the same k instead of reading it as a grade.
Myth
Hit rate and recall are the same thing.
Fact
They agree when each question needs one passage. When it needs two, hit rate says success after finding one, recall says half.
Myth
A bigger k always makes the product better.
Fact
Recall rises, but tokens, cost and distraction rise too. Choose the smallest k where recall stops improving.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
With few test questions, rates are noisy. For n questions and a rate p the standard error is about sqrt(p * (1 - p) / n): with n = 20 and p = 0.8 it is about 0.09, so a move from 0.80 to 0.85 is within noise. Do not celebrate small differences on small sets; look at which questions changed (Topic 9.7).
- ▸
Average per question (a macro average), not over pooled passages. Otherwise a question with ten relevant passages outweighs ten questions with one.
- ▸
Report the metric at the k you actually send, plus one larger k. A big gap between them means a re-ranker or a better first-stage ordering could pay off (Phase 6).
- ▸
Recall measured against a labelled golden set is a lower bound on the truth: a passage that really helps but is not labelled counts as irrelevant. This is another reason to review labels (Topic 9.2).
Remember this
- 1
Why these numbers exist. Topic 9.1 said to check retrieval first. Checking needs numbers for 'did search find what it should', computed from ids alone. Three small formulas do most of the work, and they are the vocabulary of every retrieval discussion.
- 2
The setup. For one question, the retriever returns a ranked list, best first. The golden set says which passages are relevant (answer the question). Look only at the first
kof the list, the samekyour product sends to the model. Letfoundbe how many of thosekpassages are relevant. - 3
Recall@k = found / (all relevant passages). The share of the needed passages that made it into the top k. For 'My daughter is 12, can she ride?' the needed passages are D11 (minimum age) and D4 (helmets). With top 3 = D4, D9, D11, found = 2, recall@3 = 2/2 = 1.000. If only D4 had been in the top 3, recall would be 0.500.
- 4
Precision@k = found / k. The share of the top k that was needed. Same question: 2 of the 3 passages are relevant, precision@3 = 2/3 = 0.667. Note the divisor is
k, not the length of the list: a list shorter thankis penalised. - 5
Hit rate@k = 1 if found is at least 1, else 0. Did the top k contain at least one relevant passage? Averaged over questions it is the share of questions where retrieval had a chance. It is the right number when one passage is enough to answer, and it is the simplest to explain to a non-engineer.
- 6
What the example shows. Five questions at k = 3: mean recall 0.700, mean precision 0.333, hit rate 0.800. The login question scores 0 on all three, because the right passage D3 is at rank 4, just outside the top 3. The sweep over k shows the trade: recall climbs from 0.400 at k = 1 to 1.000 at k = 4, while precision falls from 0.600 to 0.280 at k = 5.
- 7
Which one matters for RAG. Mostly recall and hit rate, because the model reads all k passages and can answer from any of them: a missing passage is a guaranteed failure, an extra passage is usually only a cost. But precision is not free. Extra passages cost tokens on every call and can distract or mislead the model (Topic 8.2), so look at recall and precision together and choose k where recall stops improving.
- 8
Reading precision correctly. If a question has one relevant passage, the best possible precision@3 is 1/3 = 0.333, so a perfect retriever scores 0.333 on that question. Precision is not a percentage to push to 100: compare it between systems at the same k. Also, unanswerable questions have no relevant passage, so recall divides by zero. Leave them out of these averages and score them with a refusal rate (Topic 9.9).
Explain it without notes
Define recall@k, precision@k and hit rate@k and compute each for a question whose relevant set is {D11, D4} and whose top 3 is D4, D9, D11.
Why does precision usually fall as k grows, while recall can only rise?
Which of the three matters most for RAG and why? Is precision ever important?
How should unanswerable questions be handled in these averages?
Practice
Write mean_recall(runs, k) where runs is a list of (ranked, relevant) pairs. Print it for two questions at k = 2.
Write smallest_k(runs, target) that returns the smallest k from 1 to 5 where mean hit rate reaches target, or None. Use the three runs in the answer.
Trade-offs
- ↔
Small k: cheap prompts and high precision, but a near miss costs the whole answer. Large k: forgiving, but costly and distracting.
- ↔
Hit rate: simple and clear for single-passage questions. Recall: fairer when answers need several passages.
- ↔
Reporting one k: easy to read. Reporting a few: shows how much headroom a re-ranker could use.
Done when you can
Done when you can compute recall@k, precision@k and hit rate@k by hand for one question.
Done when you can explain why precision has a ceiling of 1/k for single-passage questions.
Done when you can pick k from a recall and precision sweep.
Done when you can say why unanswerable questions are left out of these averages.