MRR: same hit rate, different order
Both systems return the same passages in the top 3 for four of the five questions, so hit rate at 3 is equal. Only the order differs.
Recall and hit rate ignore order. MRR rewards the right passage for being near the top, and nDCG does the same for several passages with different importance. Both are recomputed here from the formula.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write mrr(first_ranks) taking a list of ranks (0 for a miss) and print it for [1, 3, 0, 2] with 3 decimals.
Write ndcg(gains, k) for a list of gains in returned order, using log2(i + 1) and an ideal built by sorting the same gains. Print it for [1, 0, 2] and k = 3.
Explain it without notes
Compute reciprocal rank for first relevant positions 1, 2 and 5, and explain why MRR is the mean of these.
Write the formula for DCG@k and nDCG@k, and compute nDCG for gains 0, 3 at positions 1 and 2 when the ideal is 3, 0.
Why can two retrieval systems have the same hit rate and different MRR?
When would you choose nDCG over MRR?
Expected output
old first relevant at: 1, 2, 1, 4, 1
old reciprocal ranks: 1.000, 0.500, 1.000, 0.250, 1.000
old MRR 0.750 hit rate at 3 0.800
new first relevant at: 2, 3, 2, 4, 2
new reciprocal ranks: 0.500, 0.333, 0.500, 0.250, 0.500
new MRR 0.417 hit rate at 3 0.800