PHASE 0Beginner ~23 min· topic 8 of 8

Topic 0.8

Playbook: Is RAG the Right Tool?

In one line

Two recipes to start any RAG project with: a decision function that says whether to paste, retrieve or fine-tune, and a baseline harness that measures how often your retriever finds the right passage. Plus where a real embedding call plugs in.

Think of it like this

A doctor doesn't prescribe before examining, and records a baseline (weight, blood pressure) before treatment so they can tell later whether it worked. These two recipes are the examination and the baseline for a RAG project.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Baseline: The measured result of the simplest working version. Every change is compared with it.
Test question set: Real questions paired with the passage that answers them, used to measure retrieval.
Hit rate at k: The share of test questions whose right passage appears in the top k results.
Harness: Code that runs a component on many inputs and reports how it did.
Regression: A change that makes something worse that used to work.

Step by step

01Step 1: decide with reasons

recommend returns a decision and the reasons. Reasons are what reviewers check and what changes when the facts change. The example runs it on four situations from this phase.

02Step 2: write the questions before the code

Collect questions the way users really ask them: from support tickets, search logs, or by asking colleagues. Include paraphrases ("my tyre is flat" for the flat-tyre passage) and at least one question per important document.

tests.pywhole filepython

TESTS = [
    ("What does the monthly pass cost?", "D8"),
    ("I forgot my login, how do I get back in?", "D3"),
    ("Can I get my money back if I cancel?", "D1"),
]

03Step 3: measure the simplest retriever

Run the harness with word overlap. It prints, for every question, where the expected passage landed. The baseline for these six questions: 4 of 6 first (0.67), and all 6 somewhere in the top 3 (1.00).

Step 3: measure the simplest retrieverdiagram

Rendering diagram…

04Step 4: read the misses

The questions that didn't land first are a to-do list. "Can I get my money back if I cancel?" puts refunds second, behind flat tyres: the question says cancel, the passage says cancelled, and flat tyres wins on get and if. The login question puts the password passage third. Two fixes come straight out of those lines: word forms and common words (Phase 1), and meaning (Phases 2 and 3).

05Step 5: plug in a real embedder

When the toy is no longer enough, the embed call becomes an HTTP request to an embedding endpoint or a local model. Keep it behind the same function so the harness doesn't change (Topic 2.9 builds the Embedder interface).

Try it yourself

1
Add a hard question
Add ("How long do you keep things people forget?", "D9") to the tests, and add the lost-items passage D9 to the handbook. Predict the new hit rates, then run.
2
Swap the retriever
Write a second retriever that ignores the words i, my, do, in, get, the, a, is and pass it to evaluate. Did hit rate at 1 go up? Which question changed?
3
Argue with the decision
Run recommend for 200 pages that never change and need no sources. What does it say, and is that what you would do? Add a reason that changes the decision.

Code & diagrams

Recipe: a baseline harness for any retriever New tab

The retriever is a parameter. Pass a different function and the same questions measure it.

Expected output

place 1  expected D8, got D8, D10, D3  |  What does the monthly pass cost?
place 1  expected D2, got D2  |  When do shops close on public holidays?
place 1  expected D4, got D4, D1, D10  |  Is a helmet included?
place 3  expected D3, got D1, D10, D3  |  I forgot my login, how do I get back in?
place 2  expected D1, got D10, D1  |  Can I get my money back if I cancel?
place 1  expected D10, got D10, D1, D4  |  My tyre is flat, what now?
hit rate at 1: 4/6 = 0.67
hit rate at 3: 6/6 = 1.00

Recipe: paste, retrieve or fine-tune? New tab

The window size and tokens per page are example values; use your model's and your documents'.

Expected output

One 4-page policy, stable
  -> paste the documents into the prompt (small, stable, and no sources needed)
The 60-page handbook, edited weekly, sources required
  -> use RAG (the documents change often; answers must point to their source)
A 2000-page help centre
  -> use RAG (the corpus is about 1000000 tokens, more than half the context window)
Replies are correct but too formal
  -> improve the prompt, then consider fine-tuning (the problem is how the model answers, not what it knows)

Where a real embedding model plugs inpython

Not run here: it needs a provider, a key and the network. The request and response shape is the OpenAI-compatible /embeddings endpoint; check your provider's documentation for its model names, input limits and prices. Real Java projects use Jackson (or Gson) for JSON, as here.

# pip install httpx
# A real embedding call over plain HTTP. Many providers offer this OpenAI-compatible endpoint.
import os

import httpx

BASE_URL = os.environ["EMBEDDINGS_BASE_URL"]   # your provider's API root, ending in /v1
MODEL = os.environ["EMBEDDINGS_MODEL"]         # the embedding model your provider offers


def embed(texts):
    """One request for many texts. Returns one vector (a list of floats) per text, in the same order."""
    response = httpx.post(
        BASE_URL + "/embeddings",
        headers={"Authorization": "Bearer " + os.environ["EMBEDDINGS_API_KEY"]},
        json={"model": MODEL, "input": texts},
        timeout=30,
    )
    response.raise_for_status()
    items = sorted(response.json()["data"], key=lambda item: item["index"])
    return [item["embedding"] for item in items]


vectors = embed(["How do I reset my password?", "Shops open at 7 am."])
print(len(vectors), "vectors of", len(vectors[0]), "numbers each")

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Measure only the top result

In the Python tab of the harness, call evaluate(overlap_retriever, TESTS, 1) instead of 3. Run.

terminal

$ Run

── what you'll see ──

place 1 expected D8, got D8 | What does the monthly pass cost?

place 1 expected D2, got D2 | When do shops close on public holidays?

place 1 expected D4, got D4 | Is a helmet included?

MISSED expected D3, got D1 | I forgot my login, how do I get back in?

MISSED expected D1, got D10 | Can I get my money back if I cancel?

place 1 expected D10, got D10 | My tyre is flat, what now?

hit rate at 1: 4/6 = 0.67

hit rate at 3: 4/6 = 0.67

Myth vs fact

Myth

You need hundreds of test questions before measuring anything.

Fact

Twenty well-chosen real questions already catch most regressions. Start small, add every failure users report.

Myth

The model's answers are what you should measure first.

Fact

Measure retrieval first. If the right passage isn't retrieved, no answer measurement will tell you why it's wrong.

Myth

Model-written test questions are as good as real ones.

Fact

They tend to reuse the document's own words, which flatters keyword search. Use real user questions wherever you can (Topic 9.2).

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Version the test set with the code and treat a drop in hit rate like a failing test. Topic 9.7 wires it into CI.
▸
When a user reports a wrong answer, add the question and its right passage to the test set before fixing it. The set then grows exactly where the system is weak.
▸
Keep the expected answers as passage ids, not as text: ids survive rewording of the answer and let you measure retrieval without any model call, so the harness runs in milliseconds.

Remember this

1
Recipe: the decision function. When to use it: at the start of any "let's add AI over our documents" project, in the design document. What it guards against: building a retrieval system for a four-page policy, or fine-tuning to teach facts. How to adapt it: change the window size and tokens per page to your model and documents, and add your own reasons (for example, "documents are access-controlled" pushes towards RAG with filters).
2
Recipe: the baseline harness. When to use it: before you change anything about retrieval. Write 20 to 50 real questions, each with the id of the passage that answers it, and record where that passage lands. What it guards against: improvements that are really regressions, and arguments settled by whoever demoed last. How to adapt it: the retriever is passed in as a function, so the same harness measures word overlap today, BM25 in Phase 1 and vectors in Phase 3.
3
Recipe: the hit rate. Report two numbers: hit rate at 1 (the right passage was first) and hit rate at k (it was anywhere in the top k). Hit rate at k is what matters for RAG, because the model reads all k passages. Phase 9 replaces these with recall@k, MRR and nDCG, but these two are enough to start.
4
Recipe: the real embedding call. When you leave the course's examples, the toy embedder is replaced by an embedding API or a local model. The non-runnable block below shows the OpenAI-compatible /embeddings request over plain HTTP in all three languages: many texts in one request, vectors back in the same order, a timeout, the key from the environment. Topic 2.8 covers batching, limits and local models.
5
How the recipes fit together. Run the decision function first. If it says RAG, write the test questions next, before any retrieval code, and run the harness on the simplest retriever. That number is your baseline. Every later change must beat it on the same questions, and the questions it misses tell you what to build next.

Explain it without notes

What does the baseline harness measure, and why is it written before improving retrieval?

What is the difference between hit rate at 1 and hit rate at k, and which matters more for RAG?

Why should recommend return reasons as well as a decision?

Practice

Write hit_rate(results, expected, k) where results is a list of ranked id lists and expected a list of ids. Print the hit rate at 1 and at 2 for three made-up questions.

Extend recommend with a parameter access_controlled that adds the reason "documents have per-user permissions" and therefore recommends RAG. Print the decision for a 4-page document with permissions.

Trade-offs

↔
A small hand-written test set: quick and realistic, but noisy; a few questions can swing the rate a lot.
↔
Measuring retrieval only: fast and model-free, but blind to answer quality, which Phase 9 adds.
↔
A rule-based decision function: transparent and reviewable, but it simplifies; it starts the discussion rather than ending it.

Done when you can

Before you ship: you have written down why RAG (not pasting, not fine-tuning) fits this problem.
Before you ship: you have at least 20 real test questions, each with the passage that answers it.
Before you ship: you know the baseline hit rate at k of the simplest retriever.
Before you ship: refusals happen in code when nothing clears the evidence bar, with one fixed refusal sentence.
Before you ship: every answer returns its sources and their scores.

Command Palette

Words you'll meet

Step by step

01Step 1: decide with reasons

02Step 2: write the questions before the code

03Step 3: measure the simplest retriever

04Step 4: read the misses

05Step 5: plug in a real embedder

Try it yourself

Add a hard question

Swap the retriever

Argue with the decision

Code & diagrams

Break it on purpose

Measure only the top result

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can