Topic 13.6
Build: Ship AskDocs
In one line
AskDocs assembled: an access filter, keyword and vector search fused with reciprocal rank fusion, an evidence threshold, a numbered-source prompt, a grounding check, a citation check and a refusal, followed by an evaluation report that fails its launch gate first and passes it after the fix.
Think of it like this
The final test drive before a car leaves the factory. Every part was built and tested alone. Now the whole car is started, driven on a track, and a checklist decides whether it ships. This topic is that test drive for AskDocs.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- End to end
- Running the whole path from the question to the final answer, not one part alone.
- Evidence bar
- The lowest vector score at which a passage counts as an answer; below it the system refuses.
- Access filter
- The step that removes passages the user may not see before any search.
- Grounding check
- A cheap test before the model call that the best passage really shares something with the question, here at least one non-stop word.
- Stop word
- A very common word such as 'the' or 'what' that says little about the topic.
- Gate
- A check that fails the build when a metric is below its limit.
- Exit code
- The number a program returns to say whether it succeeded (0) or failed (anything else).
- Regression test
- A saved case that fails if an old problem comes back.
Step by step
01Step 1: filter by access
visible(role) returns the passages the role may see. Everything below searches only those, so the repair rota (staff only) cannot appear for a public user, however the question is worded.
02Step 2: two retrievers, one fusion
BM25 ranks by words, the toy embedder by meaning dimensions. fuse adds 1 / (60 + place) over both rankings. A passage that is high in both wins, and one found by only one retriever can still reach the top two.
03Step 3: the evidence bar
If the best vector score is under MIN_COSINE (0.5) the answer is the refusal sentence, with no model call. Questions with no word in the toy's vocabulary score zero and are refused.
04Step 4: numbered sources and the model
build_prompt lists the chosen passages as [1], [2], then the question. The fake model quotes the first sentence of source [1] and cites it.
Answer using only the numbered sources. Cite like [1].
[1] Monthly pass: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.
[2] Flat tyre: If you get a flat tyre, lock the bike, mark it as needs repair in the app and take another bike nearby at no extra cost.
Question: What does the monthly pass cost?05Step 5: verify before returning
Before the model is called, shares_content_word checks that source [1] has at least one non-stop word in common with the question. After the call, citations_ok finds every [n] in the reply and checks that each is between 1 and the number of sources, and that there is at least one. A failed grounding check, or an invalid or missing citation, becomes a refusal.
06Step 6: score it on a golden set
The report runs ten questions: seven answerable and three not. It prints hit at 1 and MRR for each retriever. Then it runs the full pipeline twice: with the grounding check off (the gate fails with exit code 1, because one unanswerable question was answered) and on (every gate passes, exit code 0). Each run shows how many answerable questions got the right source and how many unanswerable ones were refused.
Try it yourself
- 1
See what the check costs
In the first example, replace 'Do you rent scooters?' with 'What is the price of a subscription?' and run it, then set
REQUIRE_OVERLAPtoFalseand run it again. Which run answers correctly, and which refuses? Explain why the check cannot tell this question from the scooter one, and name one change that could (stemming, a synonym list, a re-ranker). - 2
Change who is asking
In the first example, change the last question's role to 'public' and then to 'staff' and note the sources. Then add a new staff-only passage and write a question that only staff can get answered.
- 3
Move the bar
Set MIN_COSINE to 0.9 and run the report. Which answerable questions are now refused? What does that say about choosing the bar from a handful of examples?
Code & diagrams
Toy embedder and fake model, so it runs identically in three languages. The structure is the real one: replace embed() and fake_model() with real calls and nothing else moves.
Expected output
Q (public): What does the monthly pass cost?
sources: D8, D10
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [1]
Q (public): I forgot my login, how do I get back in?
sources: D3, D1
answer: To reset your password, open the app, tap Forgot password and follow the link we email you. [1]
Q (public): Do you rent scooters?
sources: none
answer: I could not find that in the handbook.
Q (public): When does the repair crew swap the rack locks?
sources: D12, D10
answer: Always lock the bike to a rack with the built-in chain. [1]
Q (staff): When does the repair crew swap the rack locks?
sources: D60, D12
answer: Staff only: the repair crew swaps the rack locks on the first Monday of each month. [1]The same core with a golden set. The first run fails honestly: the toy cannot tell a scooter price from a pass price. The grounding check fixes it, and the answerable-questions gate proves the fix did not refuse good questions. With a real model, calibrate the threshold and consider a re-ranker or a judge (Topics 8.4, 6.4 and 8.7).
Expected output
golden set: 7 answerable, 3 unanswerable
keyword hit@1 5/7 mrr 0.833
vector hit@1 6/7 mrr 0.929
hybrid hit@1 7/7 mrr 1.000
before the grounding check (switch off)
answers with the right source: 7/7
refusals: 2/3
wrongly answered: What is the price of a scooter?
gate hybrid hit@1 >= 0.85: PASS
gate every answerable question answered with its source: PASS
gate every unanswerable question refused: FAIL
exit code: 1
after the grounding check (switch on)
answers with the right source: 7/7
refusals: 3/3
gate hybrid hit@1 >= 0.85: PASS
gate every answerable question answered with its source: PASS
gate every unanswerable question refused: PASS
exit code: 0Try: Press Search with the hybrid ranker, then Answer. A real embedding model finds the passages (the locking passage is first, with a cosine of about 0.75 against about 0.48 for the next) and a real chat model answers from them, with a citation. Check that the first source holds the fee, that the answer states it and cites it. Then ask a question the handbook cannot answer and see whether the refusal sentence appears.
Goals
- The locking passage is first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Answer from the sources · real model
Search first, then sign in (free) to send the retrieved passages to a real model.
- States the 150 dollar fee
- Cites a numbered source
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Apply the access filter after searching
In the Python tab of the first example, in answer, change docs = visible(role) to docs = HANDBOOK, so every passage is searched whatever the role, and run.
Myth vs fact
Myth
If every part works alone, the product works.
Fact
The parts interact: a threshold hides a retrieval gain, a filter changes the statistics of BM25. Only the end-to-end run and its report show that.
Myth
A failing gate means the build is broken.
Fact
It means the gate did its job. The failing question becomes a regression test and a task.
Myth
A fake model makes the exercise meaningless.
Fact
The structure, order and checks are the product. Swapping the model is the smallest change, and with the fake you can test every other part deterministically.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep
answerthin: it calls named functions in order. Each can be tested alone and swapped without touching the others, which is how the real service in Phase 11 stays maintainable. - ▸
Return the trace (ids, scores, whether it refused) from
answer. The report and the dashboards both need it, and with it a bad answer can be triaged in minutes (Topic 13.4). - ▸
A gate should name the failing question, not just the metric. 'Hit at 1 dropped' sends you searching; 'wrongly answered: What is the price of a scooter?' sends you to the fix.
Remember this
- 1
Why the product needs this now. Twelve phases built parts: tokenising and BM25 (Phase 1), embeddings (Phase 2), an index behind an interface (Phases 3 and 5), ingestion with metadata (Phase 4), fusion (Phase 6), query handling (Phase 7), cited and refused answers (Phase 8), an eval harness (Phase 9), guards (Phase 10) and a service shape (Phase 11). A product is the parts running together, so this topic runs them together.
- 2
What the step adds. One program, the same in Python, JavaScript and Java:
visible(role)is the access filter,bm25_rankandvector_rankare the two retrievers,fuseis reciprocal rank fusion with the constant 60,MIN_COSINEis the evidence bar,shares_content_wordis the grounding check (switched byREQUIRE_OVERLAP),build_promptnumbers the sources,fake_modelstands in for the model, andcitations_okverifies the answer.answer(question, role)wires them in the order of the query path. A second example adds the golden set and the report. - 3
How it plugs together. The order is the one from Topic 13.2: filter by access first (so the model never sees a restricted passage), search both ways, check the evidence bar on the vector score, fuse, take the top two, run the grounding check on the first one, build the prompt, call the model, verify the citations, and refuse if they are invalid. Each name matches the earlier phases, so any part can be swapped for the real thing behind the same signature.
- 4
What the first run shows. Public staff-only text is invisible: the question about the repair crew's schedule retrieves the locking passage for a public user and the staff-only passage for a staff user. 'Do you rent scooters?' is refused without a model call, because no word in it is in the toy's vocabulary. Paraphrases such as the login question work because the toy embedder maps 'login' and 'password' to the same meaning dimension.
- 5
What the report shows, before the fix. On the ten-question golden set the keyword ranker gets 5 of 7 right at first place, the vector ranker 6 of 7, and the hybrid 7 of 7 (MRR 1.000). Then the report runs the whole pipeline with the grounding check switched off, and the launch gate fails: 'What is the price of a scooter?' is answered, because the word 'price' sits in the toy's price dimension and the evidence bar cannot tell a price question about a scooter from one about a pass. Refusals are 2 of 3, exit code 1. A report that can fail is the whole point of building it.
- 6
The fix, and what it costs. The grounding check asks one cheap question before the model is called: does the best passage share at least one real word (not a stop word such as 'the' or 'what') with the question? The scooter question shares none, so it is refused. With the switch on, the report is run again: refusals 3 of 3, all seven answerable questions still answered with the right source, exit code 0. The check is general, not a patch for one question, but it is blunt. A real paraphrase with no shared word is now refused too: try 'What is the price of a subscription?', which means the monthly pass but shares no word with it. And one shared word proves little: 'fee' lets a wrong passage through. That is why the report has a gate for answerable questions as well: a stricter filter must not silently refuse good questions. Exact word matching is the crudest version; stemming (Topic 1.1), a calibrated score threshold or a re-ranker (Topics 6.4 and 8.4) are the stronger ones.
- 7
What is still missing, and how to close it. The embedder and the model are toys, so the numbers are not real-model numbers, and ten questions is a tiny golden set. Replace
embedwith a real embedding model (Topic 2.8), replacefake_modelwith a real chat call (Topic 8.1, and the recipes in the Phase 8 playbook), calibrateMIN_COSINEon a larger golden set (Topic 8.4) and add a judge-model grounding check (Topic 8.7). Keep both questions as permanent regression cases: the scooter price must stay refused, and the subscription price is a known wrong refusal to turn into an answer when the check gets smarter.
Explain it without notes
Walk through the order of steps in answer and say why each comes where it does.
Why does the access filter run before the search and not after it?
Why does the report fail its gate before the grounding check, why does it pass after, and what does the fix cost?
What would you change to move this from toy to real?
Practice
Write visible(docs, role) over a list of (id, access) pairs: public for everyone, staff only for role 'staff'. Print the visible ids for 'public' and for 'staff'.
Write exit_code(metrics, limits) that returns 1 if any metric is below its limit and 0 otherwise. Print the result for one passing and one failing set.
Trade-offs
- ↔
A fake embedder and model: deterministic, free and testable, but the numbers say nothing about real quality.
- ↔
A strict gate: safe releases and occasional blocked changes. A loose one: fast releases and silent regressions.
- ↔
One thin
answerfunction: easy to read and swap parts, with the trace as an extra return value to design.
Done when you can
Done when you can run both examples and explain every line of the first output.
Done when you can name the order of the query path and the reason for it.
Done when you can explain why the gate fails before the grounding check, why it passes after, and what the check can wrongly refuse.
Done when you can list what to replace to make AskDocs real.