AskDocs query layer: five configurations, seven questions
The retriever is the word-overlap one used in this whole phase, so effects are readable. Numbers with the Phase 6 hybrid retriever will differ: re-run the evaluation on your own system.
AskDocs gets a query layer in front of retrieve: a standalone-question step, a rewrite step and a multi-query fusion step, each a switch, measured on seven questions. Two steps together reach 7 of 7, and the third step makes one answer worse.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write hit_at_1(cases, search) where cases is a list of (question, expected_id) and search(question) returns ids best first. Return the number of cases where the first id is the expected one. Print the result for three cases and a fake search.
Write compare(cases, searches) that takes a dict name to search function and prints name hits/total for each, in the dict's order.
Explain it without notes
Why is each step a switch?
In the numbers, which steps help and which do not? What would you ship?
Why did the fused configuration miss question 5?
What belongs in a better evaluation than the seven questions here?
Expected output
baseline (raw question) hit@1 4/7 model calls 0 misses: 2, 4, 5
+ rewrite hit@1 6/7 model calls 0 misses: 4
+ standalone hit@1 5/7 model calls 2 misses: 2, 5
+ rewrite + standalone hit@1 7/7 model calls 2 misses: none
+ all three, fused hit@1 6/7 model calls 2 misses: 5