Topic 0.7
Build: The Corpus and a First Answer
In one line
AskDocs version 0: the twelve-passage handbook as typed documents, a retrieve function behind a fixed interface, a prompt builder, a stand-in model and a refusal. It answers, cites, refuses, and still gets the login question wrong.
Think of it like this
The first day of building a house is the foundation and the frame, not the kitchen. Nobody can live in it yet, but every room added later rests on it, and the walls stand where the frame says. AskDocs version 0 is the frame: every later phase adds a room without moving the walls.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Doc
- One handbook entry in the product's data: id, title and text.
- Hit
- One retrieval result: a passage with its score.
- Threshold (MIN_SCORE)
- The lowest score that still counts as evidence. Below it, a passage is ignored.
- Version 0
- The first runnable version of a product: complete end to end, simple inside.
- Stand-in model
- A fake model with the same inputs and outputs as a real one, so the rest can be built and tested first.
- Trace
- What the system did for one request (here, the sources and their scores), printed or logged so it can be checked.
Step by step
01Step 1: the corpus as typed data
Write the twelve passages as Doc values. Python uses a dataclass, Java a record, JavaScript a small factory function. The content is identical in all three tabs.
@dataclass
class Doc:
id: str
title: str
text: str
HANDBOOK = [Doc("D1", "Refunds", "A ride cancelled at least 2 hours before ..."), ...]02Step 2: retrieve behind an interface
retrieve(question, k) scores every passage, drops those under MIN_SCORE, sorts best first and keeps k. Python's sort and Java's List.sort are stable, and so is JavaScript's Array.prototype.sort, so equal scores keep handbook order in all three.
03Step 3: lay out the prompt
One instruction line, one line per source, then the question. Every source line starts with its id in square brackets, the format the model is asked to cite.
Answer using only the sources. Cite the source id in square brackets.
[D2] Opening hours: Shops open at 7 am and close at 9 pm every day. On public holidays they close at 6 pm.
Question: When do shops close on public holidays?04Step 4: answer or refuse
answer calls retrieve. No hits means the refusal constant and no model call. Otherwise it builds the prompt, calls the model and returns the reply together with the sources and their scores.
05Step 5: run four questions and read the trace
The monthly pass is found with a score of 3, but the flat-tyre passage D10 is sent too: it shares the and cost ("no extra cost"), which reaches the bar. Harmless here, a waste of tokens and a distraction at scale. The holiday hours question finds only D2, with 5 shared words. The login question retrieves refunds and flat tyres: the password passage shares only forgot and falls under the bar. The scooter question shares fewer than 2 words with every passage and is refused.
06Step 6: note what to fix next
Write the open problems down where the next phase will see them: common words like on and the count too much (Phase 1), paraphrases are invisible (Phases 2 and 3), the threshold is a guess (Phase 8). A product's backlog is part of its design.
Try it yourself
- 1
Raise the bar
Set
MIN_SCOREto 3 and run. Which questions are refused now? Was any of those refusals wrong? - 2
Ask your own question
Add
"How long does an e-bike battery last?"to the list. Predict the sources and their scores before running. - 3
Change k
Call
answer(question, 1)instead of the default 3. What disappears from the monthly pass trace, and would a real model's answer change?
Code & diagrams
The whole product as it stands after Phase 0. Each later phase's Build topic replaces the inside of retrieve() and keeps everything else.
Expected output
Q: What does the monthly pass cost?
sources: D8 (3), D10 (2)
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Q: When do shops close on public holidays?
sources: D2 (5)
answer: Shops open at 7 am and close at 9 pm every day. [D2]
Q: I forgot my login, how do I get back in?
sources: D1 (2), D10 (2)
answer: A ride cancelled at least 2 hours before its start time is refunded in full. [D1]
Q: Do you rent scooters?
sources: none
answer: I can't find that in the handbook.Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Drop the evidence bar
In the Python tab, set MIN_SCORE = 0 and run.
Myth vs fact
Myth
Version 0 should already use the best tools.
Fact
Version 0 should run end to end with clear interfaces. Then every improvement can be measured against it. Teams that start with the fanciest stack can't tell which part helped.
Myth
Sending an irrelevant extra passage is harmless.
Fact
It costs tokens on every call, and it can mislead the model, especially when it looks related. Retrieval precision matters, not only recall (Topic 9.3).
Myth
A refusal should be generated by the model.
Fact
When retrieval found nothing, code can refuse without any model call: cheaper, faster, consistent and impossible to talk out of.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Return the trace (ids and scores) from
answeritself, not only in logs. The UI can show sources, tests can assert on them, and evaluation in Phase 9 needs exactly this data. - ▸
A shared-word count is not comparable across questions: a long question naturally shares more words. That is why a fixed
MIN_SCOREmisbehaves, and why proper scoring (BM25, cosine) and calibrated thresholds replace it. - ▸
Keep
kand the threshold as configuration, not constants buried in code. They are the two knobs you will tune most, and you want to change them per deployment without a release.
Remember this
- 1
Why the product needs this now. Topics 0.1 to 0.6 showed the pieces separately. A product needs them joined with clear boundaries, so that improving one piece cannot break another. Version 0 is deliberately simple inside and deliberately careful about its edges.
- 2
The corpus as data. Each handbook entry becomes a
Docwith anid, atitleand atext. Ids are stable (D1toD12) because citations, tests and, later, deletions all refer to them. Titles are kept because they help both the model and the reader, and Topic 4.5 will show that putting titles into what gets searched improves retrieval. - 3
**The interface:
retrieve(question, k).** It returns up tokhits, each a score with itsDoc, best first, ties kept in handbook order. Inside, version 0 counts shared words. Phase 1 swaps in BM25, Phase 3 a vector index, Phase 6 a hybrid with re-ranking. None of those changes touches the code that callsretrieve. - 4
Evidence or refusal. Version 0 adds one rule from Topic 0.2: a passage only counts as evidence if it shares at least
MIN_SCORE = 2words with the question. With nothing above that bar,answerreturns the refusal constant without calling the model. That makes refusals free, instant and testable. The bar is crude, and the scooter question shows why it matters. - 5
The prompt and the model.
build_promptlists the hits as[id] title: textunder one instruction, then the question. The stand-in model answers from the first source and cites it. When Phase 8 swaps in a real model through the samebuild_prompt, the citations already have a format to follow. - 6
What is still missing. The login question is still answered wrong, now with a clear trace: the sources are
D1 (2)andD10 (2), refunds and flat tyres, each sharing two small words with the question, while the password passage shares only one and falls under the bar. Meaning-based search is Phases 2 and 3. There is no test set yet (the playbook adds one), no chunking (Phase 4), and the threshold of 2 is a guess (Phase 8 sets it properly).
Explain it without notes
What does retrieve(question, k) return, and why is it important that its shape stays fixed?
Why does version 0 refuse in code instead of asking the model?
Explain the trace for "What does the monthly pass cost?". Why is D10 included, and is that a problem?
What are the three biggest weaknesses of version 0, and which phase fixes each?
Practice
Add a function explain_answer(question) that returns the list of (doc_id, score) for every passage scoring at least 1, so a developer can see the near misses below the bar. Print it for the login question.
Change build_prompt so sources are numbered [1], [2], … and print a mapping from number to doc id beneath the prompt. When is numbering better than raw ids?
Trade-offs
- ↔
A higher evidence bar: fewer irrelevant sources and more refusals, including some wrong refusals.
- ↔
A larger k: more chances that the right passage is included, more tokens and more distraction.
- ↔
Fixed interfaces from version 0: easy to swap parts later, at the cost of a little design up front.
Done when you can
Done when you can run AskDocs v0 in all three languages and explain each line of its trace.
Done when you can say what
retrieve,build_promptandanswertake and return.Done when you can explain why refusals happen in code before the model call.
Done when you can list version 0's weaknesses and the phase that fixes each.