AskDocs harness: v0 against an improved retriever
Both retrievers are the word-overlap kind from Phase 0, so everything is exact and repeatable. The stop-word list is hand-written for this example.
AskDocs now has a scoreboard: a golden set with unanswerable rows, retrieval metrics, a refusal check, a report that compares two retrievers, and a gate that can fail a build.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write refusal_rate(golden, retrieve, k) that returns the share of rows with an empty relevant list for which retrieve returns nothing. Test it with a fake retriever that always returns ["D1"].
Write compare(old, new) taking two dicts of metrics and printing a line per metric name: old -> new (change) with 4 decimals and a sign.
Explain it without notes
What does the AskDocs harness take as input and what does it return?
Why are unanswerable rows scored separately from retrieval metrics?
Explain the output of the example: what improved, what did not, and why does the gate say FAIL?
How would you plug the answer checks of Topic 9.5 into this harness?
Expected output
metric v0 stop words change
hit rate at 3 0.6250 0.8750 +0.2500
recall at 3 0.6250 0.8125 +0.1875
MRR 0.6250 0.7917 +0.1667
refused correctly 0.5000 0.5000 +0.0000
unanswerable: Do you rent scooters? -> no sources
unanswerable: Can I bring my dog on the bike? -> D5, D7, D9
CI gate (hit rate at 3 >= 0.60 and every unanswerable refused): FAIL