Topic 7.8
Build: A Query Layer for AskDocs
In one line
AskDocs gets a query layer in front of retrieve: a standalone-question step, a rewrite step and a multi-query fusion step, each a switch, measured on seven questions. Two steps together reach 7 of 7, and the third step makes one answer worse.
Think of it like this
A restaurant adds a host at the door. The kitchen and the tables stay as they are. The host listens, fixes the order ("the usual" becomes the actual dish) and passes a clear ticket to the kitchen. Each thing the host does can be tested on its own: does the kitchen get fewer wrong tickets?
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Switch
- A setting that turns one step on or off, so its effect can be measured.
- Hit at 1
- The share of questions whose top result is the expected passage.
- Evaluation set
- Questions with known correct passages, used to compare configurations.
- Configuration
- One combination of switches.
- Regression
- A change that makes a previously correct result wrong.
Step by step
01Step 1: write the evaluation set first
Seven questions: two with typos and greetings, two follow-ups with their histories, three plain questions as a check that nothing breaks. For each, the passage that answers it.
02Step 2: write each step as a function
rewrite(question) is the rule rewrite of Topic 7.2. The standalone step is the scripted lookup of Topic 7.5. rrf(rankings) is the fusion of Topic 7.4.
03Step 3: join them in one function with switches
The order is fixed: standalone first, then rewrite, then optionally fuse the three versions of the query.
def query_layer(history, message, use_rewrite, use_standalone, use_multi, calls):
resolved = message
if use_standalone and history:
resolved = SCRIPT[message]
calls[0] += 1
cleaned = rewrite(resolved) if use_rewrite else resolved
queries = list(dict.fromkeys([message, resolved, cleaned] if use_multi else [cleaned]))
return rrf([[doc_id for doc_id, _ in retrieve(query, 3)] for query in queries])04Step 4: run the five configurations
For each, count the hits at 1, the model calls, and which questions missed. The misses column is what you read first.
05Step 5: read the misses
Baseline misses 2, 4 and 5 (the helmet typo, the pass follow-up, the late typo). Rewrite alone leaves question 4. Standalone alone leaves 2 and 5. Together they leave nothing. The fused version misses question 5 again.
06Step 6: keep what wins, switch off what does not
Ship the two steps that win. Leave multi-query off by default and re-test it when the retriever changes.
Try it yourself
- 1
Add a question that breaks it
Add a question of your own with its correct passage. Does any configuration miss it? Which step would fix it?
- 2
Drop the original from the fusion
Change the multi-query list to
[resolved, cleaned]. Does question 5 come right? What did the original query contribute? (Work out the two lists by hand first.) - 3
Count the cost
Add a counter for retrievals (one per query). How many retrievals does the all-three configuration run compared with the baseline?
Code & diagrams
The retriever is the word-overlap one used in this whole phase, so effects are readable. Numbers with the Phase 6 hybrid retriever will differ: re-run the evaluation on your own system.
Expected output
baseline (raw question) hit@1 4/7 model calls 0 misses: 2, 4, 5
+ rewrite hit@1 6/7 model calls 0 misses: 4
+ standalone hit@1 5/7 model calls 2 misses: 2, 5
+ rewrite + standalone hit@1 7/7 model calls 2 misses: none
+ all three, fused hit@1 6/7 model calls 2 misses: 5Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Rewrite the raw message instead of the standalone question
In the Python tab, make the rewrite step ignore the standalone question: change cleaned = rewrite(resolved) if use_rewrite else resolved to cleaned = rewrite(message) if use_rewrite else resolved. Run.
Myth vs fact
Myth
If each step helps alone, all steps together help more.
Fact
Steps interact. Here the third step lowers the score of a configuration that was perfect.
Myth
Seven questions prove a feature works.
Fact
They show a mechanism. A decision needs a golden set of real questions (Topic 9.2).
Myth
A query layer should always run every step.
Fact
Each step is a switch. Defaults come from measurements, not from habit.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the evaluation function in the repository and run it in CI (Topic 9.7). Query-layer changes (a new typo table entry, a prompt tweak) are exactly the changes that quietly lower hit rate on questions that used to work.
- ▸
Return the intermediate queries with the result (
message,resolved,cleaned). The UI, the logs and the tests all need them. - ▸
Latency is a budget. Each model-backed step adds its call time before any retrieval, so decide the budget first (for example 300 ms before retrieval) and see which steps fit.
Remember this
- 1
Why the product needs this now. After Phase 6 AskDocs retrieves well when the question is good. Real users send questions like those in Topic 7.1. The query layer improves the input without touching
retrieve,build_promptoranswer. Everything below the layer is unchanged. - 2
What the step adds. One function,
query_layer(history, message, switches), that returns passage ids, best first. Three steps, each a boolean switch: (1) standalone: resolve a follow-up using the history; (2) rewrite: lower-case, drop greetings, fix typos; (3) multi-query: search the raw message, the standalone question and the cleaned one, and fuse the lists with RRF. - 3
How it plugs in.
answer(question)used to callretrieve(question, k). It now callsquery_layer(history, message, ...)and then the sameretrievefor each query it produced. The filters of Topic 7.7 and the router of Topic 7.6 are also steps of this layer: the router chooses whetherretrieveruns at all, and the filter becomes an argument to it. The example leaves them out so the three switches stay easy to read. - 4
What the example measures. Seven questions with the passage that should answer each. Five configurations: nothing on, rewrite only, standalone only, both, and all three fused. Metric: hit at 1, the share of questions where the top passage is the expected one. Phase 9 builds a proper evaluation harness. This is the small version of the same idea.
- 5
What the numbers say. The baseline gets 4 of 7. Rewrite alone fixes the typo and greeting questions (6 of 7). Standalone alone fixes the two follow-ups (5 of 7). Both together get 7 of 7, with 2 model calls (one per follow-up; the no-history questions make none). Adding multi-query fusion on top goes down to 6 of 7. Question 5, "charged if i am lte", now retrieves D5 (a damage passage with "charged") first. The raw question "charged if i am lte" matches only the word "charged", which both D5 and D6 contain, so its list is D5 then D6. The cleaned question "charged if i am late" puts D6 first, then D5. Each passage is first in one list and second in the other, so RRF gives them exactly the same score (1/61 + 1/62), and the tie goes to D5 because it was seen first. Fusion cannot choose when the two searches disagree like this.
- 6
Read this result carefully. It does not say multi-query is bad. It says that on these seven short questions, with a word-matching retriever, it does not help and it costs extra searches. With a hybrid retriever and long questions the picture can reverse. The only valid decision is the one made on your own questions: turn the switch on, measure, keep it if it wins.
- 7
What is still missing. The model in the example is scripted, and real models arrive in Phase 8. Seven questions are a demonstration, not a golden set (Topic 9.2). The router and filters are shown in their own topics but not wired into the same function. Phase 8 builds the prompt and answer on top of the retrieval this layer produces.
Explain it without notes
Why is each step a switch?
In the numbers, which steps help and which do not? What would you ship?
Why did the fused configuration miss question 5?
What belongs in a better evaluation than the seven questions here?
Practice
Write hit_at_1(cases, search) where cases is a list of (question, expected_id) and search(question) returns ids best first. Return the number of cases where the first id is the expected one. Print the result for three cases and a fake search.
Write compare(cases, searches) that takes a dict name to search function and prints name hits/total for each, in the dict's order.
Trade-offs
- ↔
More steps: more problems fixed, and more latency, cost and interactions to test.
- ↔
Defaults off: safe and cheap, but you must remember to measure and switch on.
- ↔
A small evaluation set: fast to run and easy to read, but easy to overfit.
Done when you can
Done when you can explain the layer's inputs and outputs and what stays unchanged below it.
Done when you can run five configurations and read the misses column.
Done when you can say why the fused version missed question 5.
Done when you can add a question to the evaluation and see which step handles it.