Command Palette

Search for a command to run...

Hectal
Phase 9: Evaluating RAG
RAG and Vector Search · Exercise 3 of 5 PreviousNext
Advanced

nDCG with graded relevance

Write dcg(gains) where gains is a list of numbers in the order the passages were returned. It returns the sum of gain / log2(position + 1) with positions starting at 1.

Write ndcg(ranked, grades, k) where grades is a dict from passage id to gain (missing ids have gain 0). It returns dcg of the top k divided by dcg of the best possible ordering of the grades (highest first, also cut to k). If the ideal is 0, return 0.0.

Use math.log(position + 1) / math.log(2) for the discount.

What Check tests

  1. dcg discounts lower positions
  2. The ideal ordering scores 1
  3. A swapped ordering scores less
  4. Unknown ids have gain 0
  5. No relevant passage gives 0.0, not an error

Stuck? One hint at a time

The solution unlocks after your first Check. Try first: that is where the learning happens.

main.py
Sign in to run your code.

Output

Press Run to see what your program prints.