Command Palette

Search for a command to run...

Hectal
Phase 9Advanced10 of 14 in RAG and Vector Search

Evaluating RAG

Golden sets, recall, MRR and nDCG by hand, answer checks and judges, failure triage, regression tests in CI and live signals, ending in an eval harness for AskDocs.

A RAG product can fail in the search or in the writing, and a good-looking answer hides both. This phase teaches you to measure each part on its own: a golden set of real questions (including ones the handbook cannot answer), retrieval metrics computed by hand, checks on the answers, and a triage that names the stage that broke.

You will build every metric from its formula (recall, precision, hit rate, MRR and nDCG), run each in Python, Java and JavaScript with identical output, and see why two systems with the same hit rate can differ in order. You will also write rule checks and a judge-model call, learn their limits, and turn a quality bar into a regression test that can fail a build.

The Build topic gives AskDocs a scoreboard: a golden set, a harness that compares two retrievers and a gate. The playbook collects the recipes to copy into any project. LLM Engineering Phase 9 teaches evals for any model app; this phase goes deeper on what is specific to retrieval.

0/10 · 0%
10 topics ~4 h 19 code blocks & diagrams
Start with the first topic
1
9.1

Two Systems, Two Ways to Fail

A RAG product is a search system feeding a writing system, and each can fail alone. Reading a few answers cannot tell which one broke, so this phase measures retrieval and answers separately, with numbers.

22 min 1 code practice

2
9.2

A Golden Set for Retrieval

A golden set is a list of real questions with the passages that answer them, including questions the handbook cannot answer. Its quality caps the quality of every number you compute from it.

27 min 3 code practice

3
9.3

Recall@k, Precision@k and Hit Rate

Three simple numbers say how well retrieval found the right passages in its top k. Each is computed here by hand, and the sweep over k shows which one matters for RAG and why.

24 min 2 code practice

4
9.4

MRR and nDCG by Hand

Recall and hit rate ignore order. MRR rewards the right passage for being near the top, and nDCG does the same for several passages with different importance. Both are recomputed here from the formula.

22 min 2 code practice

5
9.5

Judging the Answer

Once retrieval is measured, the answer needs checking too: is it faithful to the sources, does it address the question, and did the product refuse when it should? Cheap rule checks come first, a judge model second.

27 min 3 code practice

6
9.6

Finding Where It Failed

A wrong answer can come from five different stages. A triage walks the stages in order and names the first one that broke, so each failure goes to the right fix instead of to guesswork.

24 min 2 code practice

7
9.7

Regression Tests in CI

A regression test runs the golden set on every change and fails the build when a metric falls below its floor or drops too far from the saved baseline. It turns 'we think it is better' into a check a machine enforces.

21 min 2 code practice

8
9.8

Evaluating in Production

A golden set is made of yesterday's questions. Real users bring new ones, so the product also needs signals from live use: thumbs with reasons, how often it refuses, which sources get opened, and a sample of real traffic that feeds the golden set.

22 min 1 code practice

9
9.9

Build: An Eval Harness for AskDocs

AskDocs now has a scoreboard: a golden set with unanswerable rows, retrieval metrics, a refusal check, a report that compares two retrievers, and a gate that can fail a build.

24 min 1 code practice

10
9.10

Playbook: RAG Evaluation Recipes

Recipes to copy into any RAG project: the metric functions, a triage table, a report that points at the worst questions, the CI gate, and a judge call over plain HTTP. Plus a before-you-ship checklist.

21 min 2 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Recall, precision and hit rate

Write recall_at_k(ranked, relevant, k), precision_at_k(ranked, relevant, k) and hit_at_k(ranked, relevant, k). ranked is a list of passage ids, best first; relevant is a set of ids that answer the question.

Intermediate
2

Mean reciprocal rank

Write first_rank(ranked, relevant) that returns the position (starting at 1) of the first id in ranked that is in relevant, or 0 if none is.

Intermediate
3

nDCG with graded relevance

Write dcg(gains) where gains is a list of numbers in the order the passages were returned. It returns the sum of gain / log2(position + 1) with positions starting at 1.

Advanced
4

Name the stage that broke

Write triage(expected, indexed, ranking, k, sent, answer, fact) that returns the first broken stage as one of these strings: not indexed, not retrieved, ranked too low, cut by budget, ignored by the model, or ok.

Advanced
5

A regression gate with a floor and a maximum drop

Write gate(metrics_old, metrics_new, floors, max_drop). The first three arguments are dicts from metric name to value, and max_drop is a number.

Advanced