PHASE 9Advanced ~22 min· topic 4 of 10

Topic 9.4

MRR and nDCG by Hand

In one line

Recall and hit rate ignore order. MRR rewards the right passage for being near the top, and nDCG does the same for several passages with different importance. Both are recomputed here from the formula.

Think of it like this

You look for your name on a printed list. If it is the first line you see it at once. If it is on line 40 you may give up before reaching it. Being on the list is not enough; being near the top is what counts. Hit rate only asks 'is my name on the list'. MRR and nDCG ask 'how far down is it'.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Rank (position): Where a passage sits in the result list, counting from 1.
Reciprocal rank: 1 divided by the rank of the first relevant passage, or 0 if none.
MRR: Mean reciprocal rank: the average of reciprocal rank over all questions.
Gain (grade): How useful a passage is for a question, as a number such as 0, 2 or 3.
Discount: The divisor log2(position + 1) that shrinks credit for lower positions.
DCG: Discounted cumulative gain: the sum of gain divided by discount over the top k.
nDCG: DCG divided by the DCG of the best possible ordering, so it lies between 0 and 1.

Step by step

01Step 1: find the first relevant position

Walk down the ranked list and stop at the first id in the relevant set. Return 0 if the end is reached with no match.

mrr.pywhole filepython

def first_rank(relevant, ranked):
    for position, doc_id in enumerate(ranked, start=1):
        if doc_id in relevant:
            return position
    return 0

02Step 2: turn it into a reciprocal and average

Reciprocal rank is 1 divided by the position, or 0 for a miss. The mean over questions is MRR. For the old system the reciprocal ranks are 1.000, 0.500, 1.000, 0.250, 1.000: sum 3.75, divided by 5 gives 0.750.

03Step 3: compare two systems with the same hit rate

Both systems find the right passage in the top 3 for 4 of 5 questions, hit rate 0.800. Their MRR is 0.750 and 0.417. Order is the whole difference.

Step 3: compare two systems with the same hit ratediagram

Rendering diagram…

04Step 4: give passages grades and discount lower positions

Write the gain of each returned passage in order, then divide each by log2(position + 1) and add. The example prints the gains next to each ordering so you can follow the arithmetic: 2/1 + 0/1.585 + 3/2 = 3.5.

ndcg.pywhole filepython

import math

def dcg(gains):
    return sum(gain / (math.log(position + 1) / math.log(2))
               for position, gain in enumerate(gains, start=1))

05Step 5: divide by the ideal

Sort the gains from high to low to get the best possible DCG, then divide. Because the ideal ordering scores 1.0000 against itself, nDCG tells you how far from perfect the actual ordering is.

06Step 6: choose a metric for the question

One passage is enough: MRR. Several passages or different importance: nDCG. Just a yes/no for whether search had a chance: hit rate. Report more than one, and name the formula.

Try it yourself

1
Hand-compute the swapped case
Write down the gains 2, 0, 3, 0, 0, divide each by 1, 1.585, 2, 2.322, 2.585 and add. Do you get 3.5? Then divide by 4.2619.
2
Make the new system better
In NEW, move D8 to rank 1 in the first question. What happens to MRR, and does hit rate change?
3
Change the grades
Set the gain of D4 to 1 instead of 2 and rerun. How do the DCG, ideal and nDCG values change, and why does the ideal change too?
4
A miss
Remove D11 from every ranking so it is absent. What does the swapped ordering score now? Why can nDCG be below 1.0 even for the best ordering you could return from what you retrieved?

Code & diagrams

MRR: same hit rate, different order New tab

Both systems return the same passages in the top 3 for four of the five questions, so hit rate at 3 is equal. Only the order differs.

Expected output

old first relevant at: 1, 2, 1, 4, 1
old reciprocal ranks:  1.000, 0.500, 1.000, 0.250, 1.000
old MRR 0.750  hit rate at 3 0.800
new first relevant at: 2, 3, 2, 4, 2
new reciprocal ranks:  0.500, 0.333, 0.500, 0.250, 0.500
new MRR 0.417  hit rate at 3 0.800

nDCG with graded relevance New tab

Gains: D11 = 3, D4 = 2, everything else 0. The discount is log2(position + 1), written as log(x) / log(2) so all three languages agree to the last digits shown.

Expected output

good order  gains 3, 2, 0, 0, 0  DCG 4.2619  ideal 4.2619  nDCG 1.0000
swapped     gains 2, 0, 3, 0, 0  DCG 3.5000  ideal 4.2619  nDCG 0.8212
pushed down gains 0, 0, 0, 2, 3  DCG 2.0219  ideal 4.2619  nDCG 0.4744

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the discount

In the Python tab of the nDCG example, change the discount in dcg so it divides by nothing: replace gain / (math.log(position + 1) / math.log(2)) with gain. Run.

terminal

$ Run

── what you'll see ──

good order gains 3, 2, 0, 0, 0 DCG 5.0000 ideal 5.0000 nDCG 1.0000

swapped gains 2, 0, 3, 0, 0 DCG 5.0000 ideal 5.0000 nDCG 1.0000

pushed down gains 0, 0, 0, 2, 3 DCG 5.0000 ideal 5.0000 nDCG 1.0000

Myth vs fact

Myth

MRR and hit rate tell you the same thing.

Fact

They can disagree completely: two systems with the same hit rate at 3 had MRR 0.750 and 0.417.

Myth

nDCG is a probability or an accuracy.

Fact

It is a ratio to the best possible ordering of what you could have returned. A value of 0.82 means 82 percent of the achievable discounted gain, nothing more.

Myth

Every library computes nDCG the same way.

Fact

Gain forms and discount choices vary. Compare numbers only when the formula is the same, and write the formula next to the number.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Rank-aware metrics only matter when something downstream is rank-sensitive: a budget that cuts from the bottom, a UI that shows the top source, a re-ranker. If you always send the same top k and the model reads them equally, recall@k is closer to what users see.
▸
The log2 discount is a convention, not a law of nature. It is a smooth way to say 'higher is worth more'. What matters is using the same one across experiments.
▸
Attention inside a long prompt is not uniform: models often use the first and last passages better than the middle (Topic 8.2). That is a practical reason to care about order within the top k, and the order you pack passages in is itself something to evaluate.
▸
To test a re-ranker, compare MRR and nDCG before and after on the same candidate lists. Recall@k of the candidates is its ceiling: a re-ranker cannot rank a passage that was never retrieved.

Remember this

1
Why order matters in RAG. Hit rate at 3 treats a right passage at rank 1 and one at rank 3 as equal. But order changes real outcomes: when the context budget cuts passages from the bottom (Topic 8.2), when a model pays more attention to the start of the prompt, when the interface shows only the top source, and when a re-ranker (Phase 6) is supposed to move good passages up. If you cannot measure order you cannot tell whether a re-ranker helped.
2
Reciprocal rank (RR). Find the position of the first relevant passage. RR = 1 / position, or 0 if none appears. Position 1 gives 1.000, position 2 gives 0.500, position 3 gives 0.333, position 4 gives 0.250. MRR (mean reciprocal rank) is the average of RR over all questions. It answers 'on average, how close to the top is the first good passage?'
3
The example. Two systems have the same hit rate at 3 (0.800), yet MRR separates them. In the old system the first relevant passage sits at ranks 1, 2, 1, 4, 1, so MRR is 0.750. In the new one it sits at ranks 2, 3, 2, 4, 2, so MRR is 0.417. Hit rate saw no difference at all; the new system pushed every right answer down.
4
nDCG: several passages, different importance. When some passages matter more than others, give each a gain (a grade): for example 3 for the passage that answers the question, 2 for one that is also needed, 0 for others. DCG@k = the sum over positions i = 1 to k of gain_i / log2(i + 1). The divisor, called the discount, shrinks the credit for lower positions: position 1 divides by 1, position 2 by 1.585, position 3 by 2, position 4 by 2.322, position 5 by 2.585.
5
Normalising. DCG depends on how many good passages exist, so divide by the ideal DCG (IDCG), which is the DCG of the best possible ordering (gains sorted from high to low). nDCG = DCG / IDCG, a number from 0 to 1. For 'My daughter is 12, can she ride?', with D11 = 3 and D4 = 2: the ideal order D11, D4 gives 3/1 + 2/1.585 = 4.2619, so nDCG = 1.0000. The order D4, D9, D11 gives 2/1 + 0 + 3/2 = 3.5000, so nDCG = 0.8212. Burying both at ranks 4 and 5 gives 2.0219 and nDCG = 0.4744.
6
MRR or nDCG. MRR looks only at the first relevant passage, so it suits questions where one passage is enough. nDCG looks at the whole top k and uses grades, so it suits questions that need several passages or where some passages are better than others. With only 0 and 1 as grades it is still useful: binary nDCG rewards every relevant passage for being high.
7
Cautions. There are two common forms of the gain: the one used here (gain divided by log2(i + 1)) and a variant that uses 2^gain - 1. Numbers from different forms are not comparable, so write down which you used. A question with no relevant passage has IDCG = 0 and nDCG is undefined: leave unanswerable questions out. And as with every metric, compare two systems on the same golden set.

Explain it without notes

Compute reciprocal rank for first relevant positions 1, 2 and 5, and explain why MRR is the mean of these.

Write the formula for DCG@k and nDCG@k, and compute nDCG for gains 0, 3 at positions 1 and 2 when the ideal is 3, 0.

Why can two retrieval systems have the same hit rate and different MRR?

When would you choose nDCG over MRR?

Practice

Write mrr(first_ranks) taking a list of ranks (0 for a miss) and print it for [1, 3, 0, 2] with 3 decimals.

Write ndcg(gains, k) for a list of gains in returned order, using log2(i + 1) and an ideal built by sorting the same gains. Print it for [1, 0, 2] and k = 3.

Trade-offs

↔
MRR: simple and easy to explain, ignores everything after the first relevant passage.
↔
nDCG: uses grades and the whole top k, but needs graded labels and a stated formula.
↔
A discount that falls fast (steep): focuses on the very top. A gentle one: forgiving of lower positions. Pick what your product's order sensitivity looks like.

Done when you can

Done when you can compute reciprocal rank and MRR from a list of ranks.
Done when you can compute DCG and nDCG for a list of gains by hand.
Done when you can explain a case where hit rate is equal but MRR differs.
Done when you can say when to use nDCG instead of MRR.

Command Palette

Words you'll meet

Step by step

01Step 1: find the first relevant position

02Step 2: turn it into a reciprocal and average

03Step 3: compare two systems with the same hit rate

04Step 4: give passages grades and discount lower positions

05Step 5: divide by the ideal

06Step 6: choose a metric for the question

Try it yourself

Hand-compute the swapped case

Make the new system better

Change the grades

A miss

Code & diagrams

Break it on purpose

Remove the discount

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can