AskDocs, shipped: the evaluation report
The same core with a golden set. The first run fails honestly: the toy cannot tell a scooter price from a pass price. The grounding check fixes it, and the answerable-questions gate proves the fix did not refuse good questions. With a real model, calibrate the threshold and consider a re-ranker or a judge (Topics 8.4, 6.4 and 8.7).
AskDocs assembled: an access filter, keyword and vector search fused with reciprocal rank fusion, an evidence threshold, a numbered-source prompt, a grounding check, a citation check and a refusal, followed by an evaluation report that fails its launch gate first and passes it after the fix.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write visible(docs, role) over a list of (id, access) pairs: public for everyone, staff only for role 'staff'. Print the visible ids for 'public' and for 'staff'.
Write exit_code(metrics, limits) that returns 1 if any metric is below its limit and 0 otherwise. Print the result for one passing and one failing set.
Explain it without notes
Walk through the order of steps in answer and say why each comes where it does.
Why does the access filter run before the search and not after it?
Why does the report fail its gate before the grounding check, why does it pass after, and what does the fix cost?
What would you change to move this from toy to real?
Expected output
golden set: 7 answerable, 3 unanswerable
keyword hit@1 5/7 mrr 0.833
vector hit@1 6/7 mrr 0.929
hybrid hit@1 7/7 mrr 1.000
before the grounding check (switch off)
answers with the right source: 7/7
refusals: 2/3
wrongly answered: What is the price of a scooter?
gate hybrid hit@1 >= 0.85: PASS
gate every answerable question answered with its source: PASS
gate every unanswerable question refused: FAIL
exit code: 1
after the grounding check (switch on)
answers with the right source: 7/7
refusals: 3/3
gate hybrid hit@1 >= 0.85: PASS
gate every answerable question answered with its source: PASS
gate every unanswerable question refused: PASS
exit code: 0