Recipe: metric functions and a Markdown report
Grades: 1 means the passage answers the question, 2 or 3 mean it matters more. The login question scores zero everywhere: the right passage D3 is not in the top 3.
Recipes to copy into any RAG project: the metric functions, a triage table, a report that points at the worst questions, the CI gate, and a judge call over plain HTTP. Plus a before-you-ship checklist.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write worst(table, name, n) taking a dict of metric lists and returning the indices of the n lowest values of one metric, lowest first, ties by index. Print it for {"nDCG": [0.9, 0.0, 0.5, 0.0]} with n = 2.
Write markdown_row(label, values) returning | label | v1 | v2 | with 3 decimals. Print one for "mean" and [0.75, 0.333].
Explain it without notes
What does each recipe in this playbook protect against?
Why do the metric functions take graded relevance as a dict?
Why print a look-first list in addition to the mean?
In what order would you run the checks before a release?
Expected output
| question | recall@3 | precision@3 | RR | nDCG@3 |
| monthly pass price | 1.000 | 0.333 | 1.000 | 1.000 |
| money back on cancel | 1.000 | 0.333 | 0.500 | 0.631 |
| daughter is 12 | 1.000 | 0.667 | 1.000 | 0.821 |
| login help | 0.000 | 0.000 | 0.000 | 0.000 |
| mean | 0.750 | 0.333 | 0.625 | 0.613 |
look first at: login help, money back on cancel