PHASE 8Intermediate ~33 min· topic 8 of 8

Topic 8.8

Playbook: Minimal RAG Service

In one line

Five recipes to copy: a small service that chunks, indexes, retrieves, cites and refuses; a retrieval test set that measures hit rate and picks the refusal threshold from data; an incremental re-index; a prompt template; and the before-you-ship checklist.

Think of it like this

A small library desk run by one careful librarian. They file every new document on arrival, check the shelves against the catalogue each morning (only re-filing what changed), fetch the best two pages for a question, tell you which page each fact came from, and say "we don't have that" when they don't, instead of guessing. The recipes below are that desk, in code.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Hit rate (hit@k): The share of test questions whose right document is among the first k results.
Test set: A fixed list of questions with known right answers, used to measure a change.
Fingerprint: A short number computed from a text, so a changed text can be detected without comparing everything.
Incremental update: Updating only what changed instead of rebuilding everything.
Guard: A check that stops a bad answer from reaching the user.
Stale chunk: A chunk made from an old version of a document that is still in the index.

Step by step

01Step 1: the service in one class

MiniRag has three methods: ingest(source, text), retrieve(question) and ask(question). ingest cuts a document into sentence-packed chunks with the code from Topic 8.4 and embeds each; retrieve is the top-k with a minimum score from Topic 8.3; ask builds the numbered prompt from Topic 8.5 and applies the guards from Topic 8.6.

Step 1: the service in one classdiagram

Rendering diagram…

02Step 2: run five questions through it

The first example ingests the eight documents and asks five questions. One is answered with a source, one finds no evidence and is refused without a model call, and one is answered by a model that invents a price, which the citation check catches and rejects.

03Step 3: measure retrieval on its own

The second example runs a labelled test set through retrieval only, with no model. It prints hit@1 and hit@3, shows the questions that missed at rank 1, and sweeps three thresholds against the unanswerable questions.

04Step 4: keep the index in step with the documents

The third example syncs three times: the first run embeds everything, the second finds nothing changed and embeds nothing, and the third notices a changed document, a removed one and a new one.

05Step 5: plug in the real model

Only two functions change: embed and the model call. The code block shows the model call over an OpenAI-compatible endpoint in three languages. Everything else (chunking, ranking, guards, tests) stays the same, which is the reason to build it by hand first.

Try it yourself

1
Break a guard on purpose
In the service example, make check always return an empty string. Run it and read the answer for "how much is a lock". Then put it back and see the rejection again.
2
Add a hard question
Add ("is the frame warranty longer than the battery one", "warranty") to the test set. Does retrieval put the warranty document first? Add it as a regression test you keep.
3
Watch the sync
In the sync example, add a fourth run that deletes shipping and changes hours. Predict the log lines and the final count of embedding calls before you run it.
4
Use the lab
Open the two labs in Topic 8.6 and use the numbered-sources prompt there as your template. Does the real model refuse the lock question the way the guards in code would?

Code & diagrams

Recipe: a minimal RAG service with two guards New tab

The fake model copies source [1], except for one question where it invents a price on purpose.

Expected output

ingested 8 chunks
question: get my money back
  status: ok
  answer: Unused bikes can be returned within 30 days with the receipt for a full refund. [1]
  sources: returns
question: how much is express delivery
  status: ok
  answer: Standard shipping takes 3 to 5 days. Express shipping takes 1 to 2 days and costs 12 dollars. [1]
  sources: shipping
question: how much is a lock
  status: rejected: number 20 is not in the cited source
  answer: Not in the documents.
  sources: none
question: what is the weather today
  status: refused: no evidence
  answer: Not in the documents.
  sources: none
question: does the battery come with a guarantee
  status: ok
  answer: Frames carry a 5 year warranty. Batteries carry a 2 year warranty. [1]
  sources: warranty

Recipe: a retrieval test set and a threshold from data New tab

Retrieval only, no model. Uses the fake embedder from Topic 8.3.

Expected output

top-1 miss: 'is shipping covered by the warranty' gave warranty, wanted shipping
hit@1: 10 of 11
hit@3: 11 of 11
threshold 0.20: answers 11 of 11 answerable, wrongly answers 1 of 4 unanswerable
threshold 0.50: answers 11 of 11 answerable, wrongly answers 1 of 4 unanswerable
threshold 0.80: answers 8 of 11 answerable, wrongly answers 0 of 4 unanswerable

Recipe: re-index only what changed New tab

A small polynomial hash stands in for SHA-256 so all three languages print the same numbers.

Expected output

sync 1:
  returns: added, embedded
  shipping: added, embedded
  hours: added, embedded
  embedding calls so far: 3
sync 2:
  returns: unchanged, skipped
  shipping: unchanged, skipped
  hours: unchanged, skipped
  embedding calls so far: 3
sync 3:
  returns: changed, embedded again
  shipping: unchanged, skipped
  helmets: added, embedded
  hours: removed
  embedding calls so far: 5

Where the real model plugs inpython

Not run here: it needs a network call and your own GROQ_API_KEY. The prompt (with numbered sources) is built by MiniRag, exactly as in the fake version.

# pip install openai      (Groq, and many other providers, speak the OpenAI-compatible API)
import os

from openai import OpenAI

client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.environ["GROQ_API_KEY"], timeout=30)


def real_model(prompt):
    """Drop-in replacement for fake_model in the MiniRag class: the prompt already holds the numbered sources."""
    r = client.chat.completions.create(
        model="openai/gpt-oss-20b",  # a lab model; a reasoning model, so leave room for hidden reasoning tokens
        messages=[{"role": "user", "content": prompt}],
        max_completion_tokens=600,  # Groq lists plain max_tokens as deprecated
        temperature=0,
    )
    if r.choices[0].finish_reason != "stop":
        return "Not in the documents."  # a cut-off or filtered reply is never shown as an answer
    return r.choices[0].message.content or ""


# rag = MiniRag(real_model)   # embed() must also become a real embedding call (see Topic 8.3)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the citation guard

In the service example, make ask skip the check: replace the line that sets problem (problem = check(reply, [c["text"] for s, c in hits])) with problem = "".

terminal

$ Run

── what you'll see ──

ingested 8 chunks

question: get my money back

status: ok

answer: Unused bikes can be returned within 30 days with the receipt for a full refund. [1]

sources: returns

question: how much is express delivery

status: ok

answer: Standard shipping takes 3 to 5 days. Express shipping takes 1 to 2 days and costs 12 dollars. [1]

sources: shipping

question: how much is a lock

status: ok

answer: Locks cost 20 dollars [1].

sources: shipping

question: what is the weather today

status: refused: no evidence

answer: Not in the documents.

sources: none

question: does the battery come with a guarantee

status: ok

answer: Frames carry a 5 year warranty. Batteries carry a 2 year warranty. [1]

sources: warranty

Myth vs fact

Myth

If retrieval looks good in a demo, it is good.

Fact

A handful of questions hides misses. A labelled test set with hit@k and unanswerable questions shows them.

Myth

Re-embedding everything on each deploy is simpler and fine.

Fact

It is fine for eight documents. At scale it costs money and time on every release, and replacing all chunks at once can briefly leave search empty.

Myth

A refusal is an error to be minimised at all costs.

Fact

A refusal on an unanswerable question is a correct result. Push the wrong-answer rate down first, then recover legitimate answers with better documents and chunking.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Version everything that affects results: the chunker settings, the embedding model name, the prompt template and the threshold. When quality changes, you can tell which of them moved.
▸
Swapping the embedding model changes every score, so the threshold must be re-chosen from the test set. Never reuse a threshold across models.
▸
Replacing a changed document's chunks should be atomic from the search's point of view (insert the new ones, then delete the old ones), so a query never sees zero or both versions.
▸
The test set grows from production: add every wrong answer and every refused-but-answerable question you find in the logs. After a few months it is your best asset (Phase 9).

Remember this

1
Recipe: the minimal service. When to use it: a first RAG feature over a few thousand chunks. What it guards against: wrong or invented answers. It has one class with ingest (chunk and embed), retrieve (top k above a minimum score), and ask, which has two guards: if retrieval finds nothing it refuses without calling the model, and if the model's reply fails the citation check it is replaced with the refusal. How to adapt it: swap embed for a real embedding model, fake_model for a real chat call (shown below), keep the interface of both, and keep the guards.
2
Recipe: the retrieval test set. When to use it: before you tune anything, and after every change to chunking, the embedder or the threshold. What it guards against: silent retrieval regressions and guessed thresholds. Keep 20 to 100 real questions with the document that should come first, plus a group of unanswerable questions. Measure hit@1 (the right document is first) and hit@k (it is among the first k). Then sweep the threshold and read the trade: answerable questions answered versus unanswerable ones wrongly answered. How to adapt it: label by hand from real user questions, store the set in version control, run it in CI (Topic 9.5).
3
Recipe: incremental re-index. When to use it: documents change over time. What it guards against: paying to re-embed everything on every deploy, and stale or duplicate chunks. Store a fingerprint of each document's text; on every sync embed only added or changed documents, skip unchanged ones, and delete removed ones. How to adapt it: use a real content hash (SHA-256), keep the embedding model name beside each vector and re-embed everything when it changes, and delete by source so a changed document leaves no old chunks behind.
4
Recipe: the prompt template. When to use it: always. Instructions in your template (answer only from the sources, cite like [1], one fixed refusal sentence), sources numbered from 1, the question last. What it guards against: model drift into guessing, and untrusted text overriding your instructions. How to adapt it: put the instructions in a system message when your model supports it, keep the template in one place and under version control (Topic 3.8), and never concatenate a retrieved chunk into the instructions part.
5
Recipe: the guards, in order. (1) No evidence above the threshold: refuse, no model call. (2) The model replies: reject if there is no citation, a citation to a source that was not sent, or numbers that are not in the cited source. (3) Show the user the source name from your own metadata. (4) Log the question, the retrieved ids and scores, the prompt version, the reply and the status. Count refusals and rejections: they are your best signal for missing documents and for prompt problems.
6
Before you ship. The checklist at the end of this topic is the list to walk through. The short version: test retrieval before the model; pick the threshold from data; make refusals first-class; verify every citation; keep ingestion incremental; and treat retrieved text as untrusted input.

Explain it without notes

What two guards sit around the model call in a minimal RAG service, and what does each catch?

How would you choose the refusal threshold, and why not pick it by feel?

Why is ingestion made incremental, and what do you store to make it so?

What do hit@1 and hit@3 tell you that a handful of demo questions do not?

Practice

Write hit_at_k(results, expected, k) where results is a list of ranked name lists and expected is a list of names; return the number of questions whose expected name is in its first k results. Print it for k = 1 and k = 2 on three questions.

Write plan_sync(old, new) where both are dictionaries from name to fingerprint. Return three sorted lists: names to embed (added or changed), names to delete (only in old), and names unchanged. Print them for one example.

Trade-offs

↔
Two guards add a little code and some refusals, and remove confident wrong answers.
↔
A bigger test set gives steadier numbers but costs labelling time; start small and grow it from production failures.
↔
Incremental sync needs fingerprints and deletes to be right, in exchange for cheaper, safer updates.

Done when you can

Done when you can ship a service that retrieves, cites and refuses, with the guards in place.
Done when retrieval has a labelled test set with hit@1, hit@k and unanswerable questions, and the threshold comes from it.
Done when every citation is verified and the user sees source names from your own metadata.
Done when ingestion is incremental, keyed by source, and records the embedding model.
Done when the prompt template, chunk settings, model names and threshold are versioned.
Done when retrieved text is treated as untrusted and every request logs question, chunk ids, scores, prompt version and status.

Command Palette

Words you'll meet

Step by step

01Step 1: the service in one class

02Step 2: run five questions through it

03Step 3: measure retrieval on its own

04Step 4: keep the index in step with the documents

05Step 5: plug in the real model

Try it yourself

Break a guard on purpose

Add a hard question

Watch the sync

Use the lab

Code & diagrams

Break it on purpose

Remove the citation guard

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can