Topic 6.8
Build: Hybrid Retrieval for AskDocs
In one line
AskDocs version 6: keyword and vector search fused with RRF, a re-rank stage behind the same retrieve interface, a near-copy filter and an evidence bar on the re-ranker's score. A small harness compares it with versions 1 and 3.
Think of it like this
The bike shop upgrades its customer desk. The old desk had one person who knew the manuals by heart (keyword search) and a new colleague who understood customers' own words (vector search). Now they work together at the front, and a senior technician reviews their top picks before anything is handed to the customer. The desk window, where questions come in and answers go out, did not move.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Evidence bar
- The lowest re-ranker score that still counts as evidence. Below it, AskDocs refuses.
- Hit
- One retrieval result in AskDocs: a passage position and its score.
- Interface
- The fixed shape of
retrieve(question, k), so its inside can change without touching callers. - Harness
- A small program that runs the same test questions through different versions and prints one table.
- Stand-in model
- The fake model that repeats the first source's first sentence, so the whole pipeline runs anywhere.
- Regression
- A change that makes something that worked before work worse.
Step by step
01Step 1: keep the interface
retrieve(question, k) returns hits, best first. Everything above it (build_prompt, answer, the harness) depends only on that shape, so Phase 6 can rebuild the inside.
02Step 2: stage one, hybrid
hybrid_ranking(question)[:FIRST_SIZE] fuses keyword and vector rankings with RRF and keeps 8 candidates.
03Step 3: stage two, re-rank and filter
rerank orders the candidates by pair score; drop_copies removes near-copies (the handbook has none, so it changes nothing here, but it protects you when documents are added); the evidence bar keeps only hits with a score of at least MIN_PAIR.
def retrieve(question, k=3):
shortlist = drop_copies(rerank(question, hybrid_ranking(question)[:FIRST_SIZE]))
hits = [(i, pair_score(question, TEXTS[i])) for i in shortlist]
return [hit for hit in hits if hit[1] >= MIN_PAIR][:k]04Step 4: run five questions and read the trace
The trace prints the hybrid top 3, then the sources with their scores, then the answer. For each question, check that the sources make sense and that the answer cites one of them.
05Step 5: compare versions in one harness
The second example scores versions 1, 3 and 6 on the same eight questions and two unanswerable ones. Because each version is just a function from a question to a ranking, adding version 7 later means adding one line.
06Step 6: write the backlog
Note what the trace exposed: weak sources pass the 0.5 bar (D10 and D7 for the theft question), a good question is refused (lost items), the scorer is a toy. Each becomes a task: real re-ranker (Topic 6.5), measured threshold (Phase 8), a bigger test set (Phase 9).
Try it yourself
- 1
Move the bar
Set
MIN_PAIRto 0.2 and run the trace. Which question is no longer refused wrongly, and which new weak sources appear? - 2
Remove a stage
Change
rerank(...)inretrieveto just the candidate list (no re-rank) and run the trace. What happens to the helmet question? - 3
Add version 7
Add a harness version that uses
FIRST_SIZE = 12inhybrid_rerank. Does anything change on these eight questions? What would you check before paying for it?
Code & diagrams
TOY pair scorer and toy vectors. The structure is the product: stage one, stage two, filter, evidence bar, refusal in code.
Expected output
Q: I forgot my login, how do I get back in?
hybrid top 3: D3 D1 D10
sources: D3 (0.90)
answer: To reset your password, open the app, tap Forgot password and follow the link we email you. [D3]
Q: What happens if someone steals the bike?
hybrid top 3: D10 D7 D5
sources: D12 (0.90), D10 (0.50), D7 (0.50)
answer: Always lock the bike to a rack with the built-in chain. [D12]
Q: What is the age limit for helmets?
hybrid top 3: D5 D1 D3
sources: D4 (0.87)
answer: A helmet is included free with every rental. [D4]
Q: Do you rent scooters?
hybrid top 3: D3 D5 D11
sources: none
answer: I can't find that in the handbook.
Q: Where do lost items go?
hybrid top 3: D9 D1 D2
sources: none
answer: I can't find that in the handbook.Toy components, eight questions: read the shape of the comparison, not the numbers.
Expected output
v1 keyword hit@1 4/8 hit@3 7/8 answers 2 of 2 unanswerable questions
v3 vector hit@1 2/8 hit@3 5/8 answers 2 of 2 unanswerable questions
v6 hybrid+rerank hit@1 8/8 hit@3 8/8 answers 0 of 2 unanswerable questionsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Drop the evidence bar
In the Python tab of the first example, set MIN_PAIR = 0.0 and run.
Myth vs fact
Myth
A better retriever means I can drop the refusal logic.
Fact
Even a perfect ranker returns something for every question. The evidence bar decides whether to answer, and it needs a score with a stable meaning.
Myth
Re-ranking scores can use the same threshold as vector scores.
Fact
Each model has its own score scale. A bar chosen for one model is meaningless for another; re-measure when you switch.
Myth
The harness result proves version 6 is better.
Fact
It proves the harness runs. Real claims need real embeddings, a real re-ranker and held-out questions (Phase 9).
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Return the trace from
answeritself, now including both stage ranks and the re-ranker score for each source. Evaluation (Phase 9) and debugging both need that data. - ▸
Keep
FIRST_SIZE,MIN_PAIRand the near-copy threshold as configuration. They are the knobs you will tune most. - ▸
Run the harness in CI. A change to chunking, an embedding model or a stop-word list shows up as a row that changes, which is how you catch regressions early.
- ▸
The re-ranker is a dependency: wrap it with a timeout and a fallback (Topic 6.9), and record in the trace when the fallback was used.
Remember this
- 1
Why the product needs this now. Version 1 (Phase 1) found exact words and missed paraphrases. Version 3 (Phase 3) found meaning and drowned exact terms. Topic 6.7 showed that putting them together and re-ranking lifts hit@1 from 4/8 (version 1) and 2/8 (version 3) to 8/8 on our small toy set. Version 6 packages that.
- 2
What the step adds. Stage one:
hybrid_ranking, keyword and vector rankings fused by RRF (Topic 6.3), cut atFIRST_SIZE = 8. Stage two:rerank, the pair scorer (a toy here, a real cross-encoder in production). Thendrop_copies(Topic 6.6). Then an evidence bar on the re-ranker's score:MIN_PAIR = 0.5. A fused RRF score has no absolute meaning (Topic 6.3), but a re-ranker's score does, so the bar moves from Phase 0's word counts to this score. - 3
How it plugs in.
retrieve(question, k)keeps its shape: a list of hits, best first. In this example a hit is a pair of position and score.build_prompt,fake_modelandanswerare the same as before apart from reading ids and texts from the handbook list;answerstill refuses in code, without calling the model, when no hit clears the bar. - 4
What the trace shows. The login question retrieves D3 (0.90) and answers correctly: Phase 0's version retrieved refunds. The theft question retrieves D12 (0.90) first, though the hybrid list alone had it sixth (D10, D7, D5 first); but D10 and D7 also pass at exactly 0.50, because the question's other content word,
bike, appears in them: the bar of 0.5 lets two weak sources through. The helmet question retrieves D4 (0.87); in the hybrid list D4 was fifth. The scooter question is refused. And "Where do lost items go?" is refused too, wrongly, because the toy scorer scores D9 only 0.25: it does not know thatlostandleftare related. A threshold is a trade-off, and Phase 8 sets it properly. - 5
The harness, before and after. The second example runs the same eight questions through version 1 (keyword), version 3 (vector) and version 6. Versions 1 and 3 get hit@1 4/8 and 2/8 and answer both unanswerable questions anyway (they have no evidence bar here); version 6 gets 8/8 and answers neither. Toy numbers, as Topic 6.7 warned, but the *shape* is the product: one harness, swappable
retrieve. - 6
What AskDocs can do now that it could not before: find exact codes and numbers *and* paraphrases in one search, put the best passage first more often, refuse in code when the re-ranker finds no evidence, and be compared with older versions by one harness. What is still missing. A real vector store and a real re-ranker behind the same functions (Phase 5, and Topics 6.5 and 6.9); query rewriting and routing (Phase 7); prompt layout, citations checks and conflicts (Phase 8); a real test set and measured thresholds (Phase 9); latency, caching and a fallback for a failing re-ranker (Phase 11 and Topic 6.9).
Explain it without notes
Describe the flow of a question through AskDocs v6.
Why is the evidence bar applied to the re-ranker's score and not to the fused RRF score?
What did the lost-items question show about the evidence bar?
Why keep the retrieve interface unchanged?
Practice
Write refuses(scores, bar) that returns True when no score in the list reaches the bar. Print it for [0.4, 0.3] and [0.4, 0.9] with bar 0.5, using the words refuse and answer.
Write version_table(rows) that takes a list of (name, hit1, total) and prints one line per version in the form name hit@1 x/y. Print it for [("v1", 4, 8), ("v6", 8, 8)].
Trade-offs
- ↔
Higher evidence bar: fewer weak sources and wrong answers, more wrong refusals.
- ↔
Bigger FIRST_SIZE: better recall for the re-ranker, more pair scores per question.
- ↔
Re-ranker dependency: better ordering, one more service to run, time out and monitor.
Done when you can
Done when you can run AskDocs v6 in all three languages and explain each line of the trace.
Done when you can say what
retrievetakes and returns, and why it stays fixed.Done when you can explain why the evidence bar sits on the re-ranker's score.
Done when you can list what v6 still lacks and which phase adds it.