Topic 9.9
Build: An Eval Harness for AskDocs
In one line
AskDocs now has a scoreboard: a golden set with unanswerable rows, retrieval metrics, a refusal check, a report that compares two retrievers, and a gate that can fail a build.
Think of it like this
A racing team builds a stopwatch and a test track before it tunes the engine. Without them every change is a feeling. AskDocs has built a lot (Phases 1 to 8); this step builds the stopwatch, so Phases 10 to 12 can be measured instead of guessed at.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Harness
- Code that runs a component on many test inputs and reports how it did.
- Retriever (as a function)
- Any function that takes a question and k and returns ranked passage ids.
- Refused correctly
- The share of unanswerable questions for which no passage cleared the evidence bar.
- Gate
- A pass or fail rule applied to the report, which can stop a change from merging.
- Side-by-side report
- The old and new numbers next to each other with the change.
- Stop words
- Very common words such as 'the' and 'my' that carry little meaning for matching.
Step by step
01Step 1: keep the interface
retrieve(question, k) returns up to k ids, best first, ties in handbook order, and nothing when no passage reaches the evidence bar. The harness only depends on that.
def make_retriever(stop, min_score):
def retrieve(question, k):
...
return [doc_id for score, doc_id in kept[:k]]
return retrieve02Step 2: write the golden set
Ten rows: eight answerable (including the daughter-is-12 question that needs D11 and D4) and two unanswerable (scooters, a dog). Every expected id is one of D1 to D12.
03Step 3: evaluate answerable and unanswerable rows differently
Answerable rows feed hit rate, recall and MRR, averaged over the eight. Unanswerable rows feed only the refusal share, averaged over the two. Putting them in one average would shrink the retrieval numbers for no reason (the break-it shows it).
04Step 4: print the two versions side by side
A line per metric with the baseline value, the new value and the signed change. Four decimals avoid hiding small moves and avoid rounding ties.
05Step 5: list the unanswerable questions that got an answer
For each unanswerable row print the ids the new retriever returned. The dog question shows D5, D7, D9: the product would answer from passages that have nothing to do with dogs.
06Step 6: apply the gate
Two conditions, both must hold. The report prints PASS or FAIL, and a real script would exit with a matching code. Here it prints FAIL because refusals are not perfect.
Try it yourself
- 1
Raise the bar for the new retriever
Change
make_retriever(STOP, 1)tomake_retriever(STOP, 2)in the Python tab. What happens to hit rate and to the refusal row? Which is the better trade? - 2
Add a question
Add
("How long can I keep a lost item?", ["D9"])to the golden set. Predict which retriever finds D9 first, then run. - 3
Change k
Set
K = 1. Which metrics change most? What does that tell you about how much the product depends on the ranking? - 4
Make the gate pass
Without touching the golden set, find a change that makes both gate conditions true. (Hint: look at which word the dog question matched on, and at what that word does for the other questions.) What did you give up?
Code & diagrams
Both retrievers are the word-overlap kind from Phase 0, so everything is exact and repeatable. The stop-word list is hand-written for this example.
Expected output
metric v0 stop words change
hit rate at 3 0.6250 0.8750 +0.2500
recall at 3 0.6250 0.8125 +0.1875
MRR 0.6250 0.7917 +0.1667
refused correctly 0.5000 0.5000 +0.0000
unanswerable: Do you rent scooters? -> no sources
unanswerable: Can I bring my dog on the bike? -> D5, D7, D9
CI gate (hit rate at 3 >= 0.60 and every unanswerable refused): FAILBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Average unanswerable rows into the retrieval metrics
In the Python tab, change the three divisions in the return line of evaluate from / answerable to / len(GOLDEN), so the two unanswerable rows are averaged into the retrieval metrics. Run.
Myth vs fact
Myth
If the new version's retrieval numbers are higher, ship it.
Fact
Look at every row of the report. Here hit rate rose 25 points and the refusal gap did not move. The gate exists to catch what the average hides.
Myth
The harness is finished when it prints numbers.
Fact
It is finished when it can stop a bad change: a gate with thresholds, a baseline and an exit code.
Myth
Each phase should have its own evaluation.
Fact
One golden set and one harness, rerun after every phase's change, show how each phase moved the same numbers.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Pass the retriever as a function and also pass
kand the thresholds as arguments. The harness then evaluates configurations, not just code versions, which is how you tunekand the evidence bar. - ▸
Store each run's report with the commit hash. A history of reports shows slow drift that no single comparison reveals.
- ▸
Keep the stop-word list out of the metrics. It is part of the retriever under test; if it changes, the harness should measure the change.
- ▸
Add the per-kind breakdown from Topic 9.2 next: it is a few lines in
evaluateand it is where paraphrase failures become visible.
Remember this
- 1
Why the product needs this now. The next phases add security filters, caches, a service and clever retrieval patterns. Each can quietly change what is retrieved. Without a harness you would find out from users. With one, every change gets a number and a before-and-after.
- 2
What the step adds. (1) A golden set of ten rows over the twelve handbook passages: eight answerable (one needs two passages) and two unanswerable. (2)
evaluate(retrieve): runs a retriever over the set and returns hit rate at 3, recall at 3, MRR and the share of unanswerable questions correctly refused. (3) A report that prints two retrievers side by side with the change. (4) A gate that exits non-zero when the new numbers fall short. - 3
How it plugs in. The harness only needs the interface that has existed since Phase 0:
retrieve(question, k)returning the hits, best first. Any later retriever (BM25, vectors, hybrid, re-ranked) can be passed in without touching the harness. The example builds its retrievers withmake_retriever(stop, min_score)so it can compare the Phase 0 version with a version that ignores common words. - 4
What the numbers say. v0 (word overlap, evidence bar 2): hit rate 0.6250, recall 0.6250, MRR 0.6250, refused correctly 0.5000. Ignoring common words with a bar of 1: hit rate 0.8750, recall 0.8125, MRR 0.7917. That is a real improvement on the answerable questions. But the refusal number stays at 0.5000: the scooter question gets no sources, yet the dog question still retrieves D5, D7 and D9, each sharing a single word such as bike. Word overlap cannot tell weak evidence from none.
- 5
The gate fails, on purpose. The rule is 'hit rate at 3 of at least 0.60 and every unanswerable question refused'. The first half passes and the second fails, so the gate prints FAIL. This is the point of a gate: an improvement in one place does not excuse a gap in another. A score threshold on vector similarity (Topic 8.4 and the lab in Topic 9.2) is the usual fix.
- 6
What AskDocs can do now that it could not before. Before this phase you could run AskDocs and read an answer. Now you can score a retriever on ten fixed questions, compare two retrievers number by number, see which unanswerable questions still got sources, and have a gate say FAIL. What is still missing. The harness scores retrieval only: the answer checks from Topic 9.5 would run on the output of
answer(question)and plug in the same way, but they are not wired in yet. There is no CI file around it (Topic 9.7 shows one), no logging of real production runs (Phase 11, Topic 11.6) and no per-kind breakdown (the pro note below). The refusal gap this report exposes is not fixed here: it needs a score threshold on vector similarity (Topic 8.4). Phase 10 adds guards you can measure with the same harness, Phase 11 the service, and each later Build should end by rerunning it.
Explain it without notes
What does the AskDocs harness take as input and what does it return?
Why are unanswerable rows scored separately from retrieval metrics?
Explain the output of the example: what improved, what did not, and why does the gate say FAIL?
How would you plug the answer checks of Topic 9.5 into this harness?
Practice
Write refusal_rate(golden, retrieve, k) that returns the share of rows with an empty relevant list for which retrieve returns nothing. Test it with a fake retriever that always returns ["D1"].
Write compare(old, new) taking two dicts of metrics and printing a line per metric name: old -> new (change) with 4 decimals and a sign.
Trade-offs
- ↔
One shared golden set: comparable across phases, but it can be overfitted. Keep a held-out slice.
- ↔
Gates that need every unanswerable row refused: strict and safe, hard to satisfy with word overlap. Real systems use calibrated score thresholds.
- ↔
Four decimals in reports: precise, noisier to read. Use fewer for humans and keep full precision in stored results.
Done when you can
Done when you can run the harness and explain every line of its report.
Done when you can pass a different retriever function and compare it with the baseline.
Done when you can explain why the gate fails while the retrieval numbers improved.
Done when you can say what the harness does not measure yet.