Topic 1.8
Build: Keyword Search for AskDocs
In one line
AskDocs version 1 swaps Phase 0's word counting for an inverted index with BM25, behind the same retrieve(question, k) interface, and a small harness measures the change.
Think of it like this
Replacing the engine of a car while leaving the body, the steering wheel and the pedals exactly as they were. The driver notices only that the car runs better. In AskDocs, the engine is retrieve. Everything that calls it (build_prompt, answer, the tests) stays untouched, which is the reason Phase 0 fixed its shape.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Interface
- The agreed shape of a function (its inputs and outputs) that other code depends on.
- Postings
- For each term, which passages contain it and how many times.
- MIN_SCORE
- The lowest BM25 score that still counts as evidence. Below it AskDocs refuses.
- Regression
- A change that makes something worse that used to work.
- Hit rate
- The share of test questions whose right passage was retrieved.
Step by step
01Step 1: keep the interface
retrieve(question, k) still returns up to k pairs of (score, Doc), best first, ties in handbook order. Nothing outside the function changes.
02Step 2: build the postings once
The Index constructor tokenises each passage's text, stores its length, and records term -> {passage position -> count}. The average length is computed at the end.
03Step 3: score only candidates
For each distinct query term, read its postings, compute its idf from the number of passages in the list, and add the term's BM25 contribution to each passage in the list. Passages that share no term are never touched.
04Step 4: threshold, sort, cut
Sort by score descending, break ties by passage position, drop scores below MIN_SCORE, keep k. If nothing is left, answer returns the refusal constant without calling the model.
05Step 5: run four questions
The pass question now returns D8 at 5.23. The scooters question is refused. Read the login question carefully: the right passage D3 is third at 2.01.
06Step 6: compare with version 0
The second example runs ten test questions through both retrievers using one report function. v1 is better at 1 and at 3, and the table lists which passages still go missing: D1, D9 and D12. Write those down; later phases must fix them.
Try it yourself
- 1
Move the bar
Set
MIN_SCORE = 2.5in the first example. Which question loses a source? Set it to 1.2: does the scooters question still get refused? Find the range of values that keeps all four answers and still refuses scooters. - 2
Ask your own
Add "How long does an e-bike battery last?" and "What is the monthly pass price?". Predict the first source and whether it clears the bar, then run.
- 3
Change k
Call
answer(question, 1). What disappears from the login trace, and what does that do to the chance that the right passage reaches a real model?
Code & diagrams
The whole product as it stands after Phase 1. Only retrieve and its Index are new compared with Phase 0.
Expected output
Q: What does the monthly pass cost?
sources: D8 (5.23), D10 (2.43)
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Q: When do shops close on public holidays?
sources: D2 (10.39)
answer: Shops open at 7 am and close at 9 pm every day. [D2]
Q: I forgot my login, how do I get back in?
sources: D1 (2.62), D10 (2.38), D3 (2.01)
answer: A ride cancelled at least 2 hours before its start time is refunded in full. [D1]
Q: Do you rent scooters?
sources: none
answer: I can't find that in the handbook.Ten test questions, each with the passage that should be found. Both retrievers apply their own evidence bar (2 shared words, BM25 of 2.0) and return at most 3 ids.
Expected output
retriever at 1 at 3 not in top 3
v0 overlap 0.50 0.60 D3, D1, D9, D12
v1 BM25 0.60 0.70 D1, D9, D12Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Remove the evidence bar
In the Python tab of the first example, change MIN_SCORE = 2.0 to MIN_SCORE = 0.0 and run.
Myth vs fact
Myth
BM25 scores can be used as confidence.
Fact
They depend on the question's length and wording. 2.0 means something different for one-word and ten-word questions, so a fixed bar is only a crude guard.
Myth
A better retriever fixes the wrong answers.
Fact
Here BM25 puts D3 third for the login question, and the stand-in model still answers from the first source. Retrieval quality, ranking and generation are separate things to measure.
Myth
Keep the old retriever's code around just in case.
Fact
Keep its measurement. The harness runs both so you can see the change; the old code can live in version control.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the retriever behind an interface so you can A/B it: run two retrievers on the same questions and diff the ids. The harness does that offline, and the same idea works on live traffic with logging (Topic 11.6).
- ▸
Return the scores with the hits and log them from day one. Phase 8's threshold, Phase 9's metrics and every debugging session rely on them.
- ▸
The
Indexis built once at start-up here. In a service it must be updated when documents change (the playbook shows add and remove) and rebuilt when you change the analysis, which means a re-index (Topic 11.8).
Remember this
- 1
Why the product needs this now. Version 0 scored passages by counting shared words, so
thecounted as much asmonthly, and a bar of 2 shared words decided what was evidence. BM25 weights rare words more and common words less. It is the first real retriever, and it is the baseline all later phases are measured against. - 2
What AskDocs can do now that it could not before. It ranks passages so rare, meaningful words count for more than
the; it returns a score with every source; and it refuses a question when no passage clears an evidence bar, so the scooters question gets "I can't find that in the handbook". Phase 0 could only count shared words. It still cannot understand paraphrases, it answers with a stand-in model, and it has no real documents: those gaps are listed at the end of this section with the phase that closes each. - 3
What the step adds. An
Indexclass that builds the postings once, with the term count of each passage and the average length, and ascoresmethod that visits only the passages containing a query term (Topic 1.2).retrievesorts those scores, drops anything belowMIN_SCORE, and returns the topkas(score, Doc)pairs: the same shape as Phase 0. - 4
The threshold. Version 0's bar was a count of shared words. BM25 scores are in different units, so the bar is now a BM25 score,
MIN_SCORE = 2.0. It separates the four sample questions: the scooters question, which the handbook never mentions, tops out at 1.09 and is refused, while the genuine answers score between 2.01 and 10.39. This is a guess that happens to work on four questions. Scores depend on the query (Topic 1.4), and Phase 8 sets the threshold from evidence. - 5
Reading the trace. The pass question returns D8 (5.23) and D10 (2.43, from
costandthe). The holidays question returns only D2 (10.39). The login question returns D1 (2.62), D10 (2.38) and the right passage D3 (2.01), only third and barely above the bar. The fake model still answers from the first source, so the answer is still wrong. The retrieval is better, not yet good. - 6
Measuring it. The harness reuses the ten questions from Topic 1.6 and runs both retrievers, the Phase 0 overlap rule and the new BM25, through the same function. Hit rate at 1 goes from 0.50 to 0.60, and at 3 from 0.60 to 0.70. The threshold costs recall: without it BM25 reached 0.90 at 3 (Topic 1.6), but the refusal rule cannot work without some bar. That tension is real, and Phase 8 handles it properly.
- 7
What is still missing. Paraphrases (
login) are still found by luck, not by meaning: Phases 2 and 3 add embeddings and a vector index. Titles are not indexed (Topic 1.5 shows the benefit, Phase 4 adds context to chunks). The handbook is twelve passages, not real documents (Phase 4). And there is no proper evaluation yet (Phase 9).
Explain it without notes
Why can AskDocs swap its retriever without touching answer?
How does the Index avoid scoring passages that cannot match?
Why did the hit rate at 3 fall from 0.90 (Topic 1.6) to 0.70 here?
Why is MIN_SCORE = 2.0 only a guess?
Practice
Add a function explain(question, doc_id) to a small BM25 setup that returns the list of (term, part) pairs for one passage, using the formula from Topic 1.4 with k1 = 1.2, b = 0.75. Test on two passages: "late fee" and "monthly pass" with the question "late".
Write refuse_if_weak(hits, min_score) that returns None when the best score is under min_score (or there are no hits) and the hits otherwise. Print the results for scores [2.43, 0.64] and [1.09] with a bar of 2.0.
Trade-offs
- ↔
A higher
MIN_SCORE: fewer irrelevant sources and more refusals, including wrong refusals. - ↔
A larger
k: more chances to include the right passage, more tokens and more distraction. - ↔
Building the index at start-up: simple and fast to query, but needs a rebuild when documents change.
Done when you can
Done when you can run AskDocs v1 in all three languages and read each line of its trace.
Done when you can say what changed from v0 and what stayed the same.
Done when you can explain why the evidence bar exists and why 2.0 is a guess.
Done when you can run the harness and read hit rate at 1 and at 3 for both retrievers.
Done when you can list the failures that remain and the phase that should fix each.