PHASE 9Advanced ~21 min· topic 10 of 10

Topic 9.10

Playbook: RAG Evaluation Recipes

In one line

Recipes to copy into any RAG project: the metric functions, a triage table, a report that points at the worst questions, the CI gate, and a judge call over plain HTTP. Plus a before-you-ship checklist.

Think of it like this

A pilot's pre-flight checklist. The checks are the same every flight, short, written down, and done in order, because the cost of forgetting one is high and the cost of doing them is small. These recipes are the pre-flight checklist for a RAG release.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pre-flight checklist: A short fixed list checked before every release.
Graded labels: Relevance written as numbers, such as 3 for the best passage, 2 for a helpful one, 0 for the rest.
Report: The printed result of a run: per-question numbers, means and the worst cases.
Artifact: A file a CI job saves for people to open later, such as the report.
Look-first list: The few worst questions in a run, the best place to start debugging.

Step by step

01Step 1: copy the metric functions

Recall, precision, reciprocal rank and nDCG each take (ranked, grades, k) or (ranked, grades). A passage is relevant when it appears in the grades dict. nDCG divides by the ideal built from the same dict.

02Step 2: compute a table

Run each metric for each question and keep the values in a dict by metric name. The same table drives the printed report, the means and the gate.

03Step 3: print Markdown

One header row, one row per question, and a mean row. Markdown renders in pull requests and is readable in a terminal.

04Step 4: name the worst questions

Sort by nDCG, lowest first, and print the first two. The sort is stable, so ties keep their order and the output is the same on every machine.

Step 4: name the worst questionsdiagram

Rendering diagram…

05Step 5: add the judge for what rules miss

Run rule checks everywhere, and the judge on a sample or on release candidates. Treat an unusable judge reply as an error that you count and watch.

06Step 6: run the checklist before every release

The checklist below is short on purpose. If it takes more than a few minutes to run, people skip it.

Try it yourself

1
Predict the look-first list
Before running, read the four rankings and decide which two questions have the lowest nDCG. Then run and check.
2
Change k
Set K = 2 and run. Which cells change? Which question's nDCG falls and why?
3
Add a grade
Make the login question's relevant set {"D3": 1, "D1": 1} (as if refunds also helped). How do its row and the mean row change?
4
Make the report CI-friendly
Add a line at the end that prints FAIL if the mean nDCG is below 0.60, else PASS. Which does it print, and does the answer match your sense of the table?

Code & diagrams

Recipe: metric functions and a Markdown report New tab

Grades: 1 means the passage answers the question, 2 or 3 mean it matters more. The login question scores zero everywhere: the right passage D3 is not in the top 3.

Expected output

| question | recall@3 | precision@3 | RR | nDCG@3 |
| monthly pass price | 1.000 | 0.333 | 1.000 | 1.000 |
| money back on cancel | 1.000 | 0.333 | 0.500 | 0.631 |
| daughter is 12 | 1.000 | 0.667 | 1.000 | 0.821 |
| login help | 0.000 | 0.000 | 0.000 | 0.000 |
| mean | 0.750 | 0.333 | 0.625 | 0.613 |
look first at: login help, money back on cancel

Recipe: a judge call over plain HTTPpython

Not run here: it needs a provider, a key and the network. The chat endpoint shape is the OpenAI-compatible /chat/completions; check your provider's documentation for model names, limits and prices. Real Java projects use Jackson (or Gson) for JSON, as here.

# pip install httpx
# A judge call over plain HTTP. Many providers offer this OpenAI-compatible chat endpoint.
import json
import os

import httpx

BASE_URL = os.environ["JUDGE_BASE_URL"]  # your provider's API root, ending in /v1
MODEL = os.environ["JUDGE_MODEL"]  # a capable chat model your provider offers
RUBRIC = 'Score 1 to 5. faithful: every claim is supported by the sources. relevant: the answer addresses the question. Reply with JSON only: {"faithful": n, "relevant": n, "reason": "..."}'


def judge(question, sources, answer):
    """Returns the scores as a dict, or None when the reply is not the JSON we asked for."""
    response = httpx.post(
        BASE_URL + "/chat/completions",
        headers={"Authorization": "Bearer " + os.environ["JUDGE_API_KEY"]},
        json={
            "model": MODEL,
            "temperature": 0,
            "max_tokens": 300,
            "messages": [
                {"role": "system", "content": RUBRIC},
                {"role": "user", "content": "Question: " + question + "\nSources: " + sources + "\nAnswer: " + answer},
            ],
        },
        timeout=60,
    )
    response.raise_for_status()
    try:
        return json.loads(response.json()["choices"][0]["message"]["content"])
    except ValueError:
        return None


print(judge("Is a helmet included?", "A helmet is included free with every rental.", "Yes, free with every rental."))

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Average only the questions that found something

In the Python tab, change the mean row so each mean divides by the number of questions with a value above zero: replace sum(table[name]) / len(RUNS) with sum(table[name]) / sum(1 for v in table[name] if v > 0). Run.

terminal

$ Run

── what you'll see ──

| monthly pass price | 1.000 | 0.333 | 1.000 | 1.000 |

| money back on cancel | 1.000 | 0.333 | 0.500 | 0.631 |

| daughter is 12 | 1.000 | 0.667 | 1.000 | 0.821 |

| login help | 0.000 | 0.000 | 0.000 | 0.000 |

| mean | 1.000 | 0.444 | 0.833 | 0.817 |

look first at: login help, money back on cancel

Myth vs fact

Myth

A recipe from this playbook works unchanged for every product.

Fact

The structure carries over; the thresholds, the golden set and the stage names are yours.

Myth

One metric is enough to gate on.

Fact

A single metric can be satisfied while another falls. Use hit or recall for coverage, an order metric for ranking, a refusal metric for unanswerable rows.

Myth

Evaluation is a phase of the project.

Fact

It is a part of the build that stays: new questions arrive, documents change and every release should be scored.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep the metric functions in one small module with tests worked by hand. If two people's numbers disagree, the argument should be about the data, not about whose formula is right.
▸
Treat the golden set like code: review changes to it, and never change a label in the same commit as a retrieval change that would benefit from the new label.
▸
Print the formula next to the number in reports (nDCG@3, log2 discount, gains 0 to 3). Months later nobody remembers which variant you used.
▸
Put the look-first list at the top of the CI comment. People read the first lines.

Remember this

1
Recipe: the metric functions. When to use it: as soon as you have a golden set. What it guards against: arguing about quality without numbers, and every team re-deriving the formulas slightly differently. How to adapt it: the functions take a ranked list of ids and a dict of grades (1 for binary labels, 2 or 3 for graded), so they handle single-passage, multi-passage and graded cases. Keep k as a parameter, and test each function on a case you worked by hand (the exercises in this phase do).
2
Recipe: the triage table. When to use it: whenever a question fails. What it guards against: fixing the wrong stage. The table in Topic 9.6 lists the five stages in order, the first check for each and the usual fix. How to adapt it: add the stage names and fixes your system really has (a cache, a query rewriter, a permission filter) in pipeline order.
3
Recipe: the report. When to use it: in every run, locally and in CI. What it guards against: averages that hide the worst questions. The example prints a table of per-question numbers, a mean row, and the two questions with the lowest nDCG as 'look first at'. How to adapt it: print Markdown, as here, so the same output works in a terminal, a pull request comment and a build artifact.
4
Recipe: the CI gate. When to use it: once the report is stable. What it guards against: regressions that merge unseen. Use both a floor and a maximum drop from the saved baseline for each metric, print the questions that changed, and exit non-zero on failure (Topic 9.7). Update the baseline only with a written reason.
5
Recipe: the judge call. When to use it: for faithfulness and relevance on a sample, after rule checks. What it guards against: unreadable free-text scores and silent failures on bad replies. The non-runnable block below shows a call over plain HTTP in all three languages: a rubric as the system message, temperature 0, a timeout, the key from the environment, and parsing that returns nothing when the reply is not JSON. Its model name comes from the environment because providers retire models.
6
How the recipes fit together. Write the golden set (9.2) and run the metric functions (9.3 and 9.4) through the report. Add rule checks (9.5). When something fails, use the triage table (9.6). Put the report behind the gate (9.7). Feed production failures back into the golden set (9.8). The AskDocs harness (9.9) is these pieces assembled for one product.

Explain it without notes

What does each recipe in this playbook protect against?

Why do the metric functions take graded relevance as a dict?

Why print a look-first list in addition to the mean?

In what order would you run the checks before a release?

Practice

Write worst(table, name, n) taking a dict of metric lists and returning the indices of the n lowest values of one metric, lowest first, ties by index. Print it for {"nDCG": [0.9, 0.0, 0.5, 0.0]} with n = 2.

Write markdown_row(label, values) returning | label | v1 | v2 | with 3 decimals. Print one for "mean" and [0.75, 0.333].

Trade-offs

↔
A shared metrics module: one definition for everyone, a little ceremony to maintain.
↔
A Markdown report: readable everywhere, less machine-friendly than JSON. Emit both: JSON for the gate, Markdown for people.
↔
A judge on a sample: affordable, noisy on rare failures. A judge on everything: complete, costly.

Done when you can

Before you ship: the golden set is versioned, validated, and includes unanswerable rows.
Before you ship: recall, MRR and nDCG at the k you send are at or above the floors and within the allowed drop from the baseline.
Before you ship: every unanswerable golden row is refused, and the rule checks (citations, numbers, refusal) pass on the answers.
Before you ship: the questions that changed since the last release have been read by a person.
Before you ship: a sample of answers has been judged, judge errors are counted, and the judge was compared with human labels.
Before you ship: production feedback is flowing back into candidate golden rows.

Command Palette

Words you'll meet

Step by step

01Step 1: copy the metric functions

02Step 2: compute a table

03Step 3: print Markdown

04Step 4: name the worst questions

05Step 5: add the judge for what rules miss

06Step 6: run the checklist before every release

Try it yourself

Predict the look-first list

Change k

Add a grade

Make the report CI-friendly

Code & diagrams

Break it on purpose

Average only the questions that found something

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can