Topic 0.6
The Product You Will Build
In one line
This course builds one product from start to finish: AskDocs, a service that answers questions from a company handbook, shows its sources and refuses when the handbook has no answer. Each phase adds one production-grade part.
Think of it like this
Learning to cook by making one dinner party properly, course by course, instead of practising chopping, boiling and baking as separate drills. Every skill you learn goes straight onto the table. This course teaches RAG the same way: every idea is added to one growing product.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Build-along
- A course where one project grows step by step, and each lesson adds to it.
- Production-grade
- Good enough to run for real users: it keeps its promises on inputs nobody tried in advance.
- Corpus
- The documents a system answers from. Here, the Lumen Bikes handbook.
- Refusal
- A deliberate "I can't find that" answer when there is no evidence.
- Interface
- The fixed shape of a function (its name, inputs and outputs) that lets you replace what is inside it.
- Access control
- Rules about who may see which documents, enforced by the system.
- Observable
- Built so that you can see what it did and why: which passages, which scores, which prompt.
Step by step
01Step 1: write down the promises
Before any search code, state what must be true of every answer: it cites its sources; every cited source was really retrieved; with nothing retrieved, it refuses. These are requirements, exactly like "the total must equal the sum of the items" in a billing system.
02Step 2: find the citations in an answer
Citations are ids in square brackets, such as [D8]. A small function walks through the answer and collects them. No model is involved: this is ordinary string handling, so it is fast, free and deterministic.
def cited_ids(answer):
ids = []
start = answer.find("[")
while start >= 0:
end = answer.find("]", start)
if end < 0:
break
ids.append(answer[start + 1 : end])
start = answer.find("[", end)
return ids03Step 3: review an answer against what was retrieved
review applies the promises in order. A refusal is always acceptable. Otherwise something must have been retrieved, at least one source must be cited, and every cited id must be among the retrieved ids. The first broken promise is reported.
04Step 4: run it on five answers
Run the example. Two answers pass. One cites [D9], a passage that was never retrieved: the model made up a citation. One states a true fact with no citation. One answers confidently with nothing retrieved. Each failure is a real behaviour of real models.
05Step 5: see where each phase plugs in
The diagram shows AskDocs as it will be at the end of the course. The top row is the ingestion path, which runs when documents change. The bottom row is the query path, which runs for every question. The number in each box is the phase that builds it.
Try it yourself
- 1
Fix an answer
Edit the third answer so it ends with
[D4]. Predict its review line, then run. - 2
Cite two sources
Add a case: the answer
"Late bikes cost 2 dollars per 15 minutes [D6]. The pass costs 25 dollars [D8]."with retrievedD6andD8. Then removeD8from the retrieved list. What does the review say? - 3
Add a promise
Add a fourth rule: an answer longer than 300 characters fails with
"FAIL: too long". Where inreviewshould the check go so that refusals still pass?
Code & diagrams
No model runs here. The five answers are examples of what a model might return; the review is plain code.
Expected output
The monthly pass costs 25 dollars. [D8]
retrieved: D8, D6 -> ok
Refunds take 5 working days. [D9]
retrieved: D1, D6 -> FAIL: cites D9, which was not retrieved
A helmet is included free with every rental.
retrieved: D4 -> FAIL: no source is cited
Yes, we rent scooters at every shop.
retrieved: nothing -> FAIL: nothing was retrieved, so it had to refuse
I can't find that in the handbook.
retrieved: nothing -> okBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust citations without checking them
In the Python tab, delete the three lines of review that build unknown and return the "which was not retrieved" failure. Run.
Myth vs fact
Myth
A RAG product is a retriever plus a prompt.
Fact
That is the demo. The product is also ingestion, evaluation, access control, freshness, monitoring and cost control. Those are most of the work and most of this course.
Myth
Quality can only be judged by reading answers.
Fact
Many qualities are checkable in code: a citation exists, it points at a retrieved passage, the answer refuses when it should. Automate those first; read answers for what remains.
Myth
Refusing is a failure.
Fact
A wrong answer costs far more trust than an honest "I can't find that". Refusing at the right times is a feature you design and measure.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Write the answer contract (citations required, refusal sentence, format) before choosing a model or a vector store. It is the part users experience, and it turns later arguments about tools into measurable questions.
- ▸
Keep the refusal text as one constant shared by the generator, the checks and the user interface. Teams that let the model phrase refusals freely can't count them, and the refusal rate is one of the most useful production metrics (Topic 9.8).
- ▸
The two paths have different engineering profiles. Ingestion is batch work: throughput, retries and idempotency matter. The query path is interactive: latency and availability matter. Many teams run them as separate services with only the index in common (Topic 11.1).
Remember this
- 1
Why one product. RAG ideas are easy to nod along to and hard to combine. Chunking affects retrieval, retrieval affects the prompt, the prompt affects citations, and all of them affect cost. You only feel those connections when the parts live in one system that you have to keep working. So each phase ends with a Build topic that adds its step to the same product, and you can run the product at every stage.
- 2
The product: AskDocs. A question-answering service for Lumen Bikes, a made-up city bike-rental company. Staff and customers ask questions in their own words. AskDocs answers from the company handbook and nothing else. The handbook starts as twelve short passages (refunds, opening hours, password reset, helmets, damage, late returns, batteries, the monthly pass, lost items, flat tyres, minimum age, locking) and grows as the course needs harder cases.
- 3
What "production-grade" means here. A demo answers the questions its author tried. A product keeps promises about *every* answer. AskDocs makes six. Cited: every answer names the passages it used. Honest: it only cites passages it actually retrieved, and it refuses when it found nothing. Measured: a test set says how often the right passage is found. Access-controlled: nobody sees text they are not allowed to see. Fresh: a handbook edit reaches answers quickly. Observable: for any answer you can see what was retrieved and why.
- 4
Promises you can check in code. The first three promises are checkable today with a few lines, with no model involved. The example below reviews five candidate answers against the passages that were retrieved for them. This habit, turning a quality you want into a check you can run, is most of what separates a RAG product from a RAG demo.
- 5
The map. Phase 1 gives AskDocs proper keyword search. Phase 2 adds embeddings and Phase 3 a vector index. Phase 4 builds ingestion, so real documents can be loaded, cleaned and chunked. Phase 5 stores everything in a vector database. Phase 6 combines keyword and vector search and re-ranks. Phase 7 improves the question before searching. Phase 8 writes grounded, cited answers. Phase 9 measures all of it. Phase 10 secures it. Phase 11 runs it as a service. Phase 12 adds advanced retrieval patterns, and Phase 13 assembles the final service and prepares you for interviews.
- 6
Names that stay the same. The product keeps the same small vocabulary in every phase and every language: a
Dochas an id, a title and a text;retrieve(question, k)returns scored hits, best first;build_prompt(question, hits)lays out the prompt;answer(question)ties them together. Later phases replace the *inside* of these functions. Their shape stays, which is what lets you swap a keyword retriever for a vector one without touching the rest.
Explain it without notes
What does AskDocs do, and what are the six promises it makes?
Why does the course keep the same function names and shapes in every phase?
Which of the promises can be checked with plain code, without a model? Give two examples.
Describe the two paths in a RAG system and when each one runs.
Practice
Write has_citation(answer) that returns True when the answer contains at least one [...] with something inside the brackets. Test it on three strings.
Write refusal_rate(answers, refusal) that returns the share of answers equal to the refusal sentence, and print it to two decimals for five sample answers.
Trade-offs
- ↔
One growing product: you see how the parts affect each other, but each phase depends on the ones before it.
- ↔
Strict promises (always cite, refuse without evidence): trustworthy answers, but more refusals and more engineering than a free-form chatbot.
- ↔
A tiny made-up handbook: every example is readable, but some problems (scale, messy PDFs) only appear with real data, so later phases add harder documents.
Done when you can
Done when you can describe AskDocs and the company it serves in two sentences.
Done when you can list the six promises and say which three are checked in this topic's example.
Done when you can name the ingestion path and the query path and what each contains.
Done when you can say which phase builds each part of the product.