PHASE 8Intermediate ~20 min· topic 9 of 9

Topic 8.9

Playbook: Grounded Answer Recipes

In one line

Five small, reusable pieces of the answer path: a prompt renderer, a context packer, a refusal gate, a citation verifier and a number checker, plus the one place a real chat call plugs in.

Think of it like this

A tool belt with the five tools you reach for on every job. You do not redesign a screwdriver each time. These recipes are the five you reach for on every RAG answer path, and each one is short enough to read in a minute.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Recipe: A small, tested function you copy into a project.
Refusal gate: A check that stops the request before the model is called.
Verifier: Code that checks a model's output without trusting it.
Overhead: Tokens used by tags and labels around each passage.
OpenAI-compatible: An API that accepts the same chat request shape as OpenAI's, offered by many providers.
Environment variable: A named setting outside the code, where secrets such as API keys belong.

Step by step

01Step 1: decide with the gate

should_answer(scores, threshold) is the first call. With no scores it returns false, so an empty retrieval can never reach the model.

02Step 2: pack and render

pack_sources keeps the passages that fit, render_sources numbers them. Together they produce the sources block of the prompt.

Step 2: pack and renderdiagram

Rendering diagram…

03Step 3: the model call

The real call is the polyShow block. It reads the key from an environment variable and returns the answer text.

04Step 4: verify before showing

verify_citations and numbers_missing return lists of problems. An empty list means show the answer; anything else means retry, flag or refuse.

05Step 5: test the recipes with fixed inputs

The demo prints one result per recipe from fixed inputs. Turn each line into a unit test: it is the whole test suite for the answer path.

06Step 6: log the outcomes

Log gate decisions, tokens used, citation problems and missing numbers. They are the early-warning signals Phases 9 and 11 build on.

Try it yourself

1
Turn the demo into tests
Rewrite the printed lines as assert statements, one per recipe. Add one failing case for each: a negative budget, an answer with [0], an answer with no number at all.
2
Swap the token rule
Change the 4-characters rule to count words instead. Which passages are packed now? How far is the word count from a real tokenizer for this text?
3
Add a recipe
Write quote_ok(quote, source) from Topic 8.6 and add it to the demo with one passing and one failing quote.

Code & diagrams

Five grounded-answer recipes New tab

Production-shaped but small. Every printed line is a unit test waiting to be written.

Expected output

gate 0.47: answer
gate 0.38: refuse
gate with no hits: refuse
packed with budget 70: 2 of 3 passages
[1] The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.
[2] Bikes returned late are charged 2 dollars for every 15 minutes after the booked end time.
verify '... [1]': ok
verify '... [3]': citation 3 is not a source
verify no marker: no citation
numbers missing: 30

Where a real chat call plugs in (OpenAI-compatible, plain HTTP)python

Shown, not run. The endpoint, model and key variable are Groq's, matching this course's labs (checked against Groq's API reference: `max_completion_tokens` replaces the deprecated `max_tokens`, and `reasoning_effort` accepts low, medium or high for this model). Groq changes its model list often, so check the current id. Other providers use the same shape with a different base URL. Java parses the reply with Jackson in a real project.

import os

import httpx

# SYSTEM, the sources block and the question come from the recipes above and Topic 8.1.
prompt = render_sources(packed) + "\n\nQuestion: " + question

response = httpx.post(
    "https://api.groq.com/openai/v1/chat/completions",
    headers={"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]},
    json={
        "model": "openai/gpt-oss-20b",
        "temperature": 0,
        "max_completion_tokens": 600,  # a reasoning model spends hidden tokens first: leave room
        "reasoning_effort": "low",
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
    },
    timeout=30,
)
response.raise_for_status()
answer = response.json()["choices"][0]["message"]["content"]

problems = verify_citations(answer, len(packed)) + ["number not in sources: " + n for n in numbers_missing(answer, packed)]
print(answer if not problems else REFUSAL)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Allow one source too many

In the Python tab, change n > n_sources to n > n_sources + 1 in verify_citations.

terminal

$ Run

── what you'll see ──

verify '... [1]': ok

verify '... [3]': ok

verify no marker: no citation

Myth vs fact

Myth

A library will handle grounding for me.

Fact

Libraries wrap the call. The threshold, the budget and the checks depend on your documents and need your measurements.

Myth

These checks slow the answer down.

Fact

They are string work on a few hundred tokens and run in microseconds. The model call is the slow part.

Myth

If the verifier says ok, ship it.

Fact

It says the citations are well formed. Evaluation (Phase 9) tells you whether the answers are right.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep the recipes in one small module with no model dependency so they can be unit tested in milliseconds and reused by the evaluation harness in Phase 9.
▸
Version the instruction text, the threshold, the budget and the embedder's name together as one configuration. A change to any of them is a change to behaviour and must pass the golden set (Topic 9.7).
▸
Never send the API key to the browser. The call in the playbook runs on your server, which also lets you rate-limit, log and cache (Phase 11).

Remember this

1
Recipe: render the sources. Use it when you build any RAG prompt. It numbers the passages from 1 and keeps the layout in one function, so the number in the answer means the same thing in your code. It guards against citations that point at the wrong passage. Adapt it by adding the delimiters, titles and dates from Topics 8.1 and 8.5.
2
Recipe: pack within a budget. Use it whenever the number of retrieved passages can vary. It walks down the ranking, charges each passage its tokens plus overhead and keeps the ones that fit. It guards against over-long prompts and surprise bills. Adapt it by swapping the 4-characters rule for your provider's tokenizer, and by choosing stop or skip on purpose (Topic 8.2).
3
Recipe: gate on a measured score. Use it before every model call. It returns False when there are no hits or the best score is below the threshold, so the model is not called. It guards against confident answers from irrelevant sources. Adapt it by loading the threshold from configuration together with the embedder's name, and re-measure it when either changes (Topic 8.4).
4
Recipe: verify the citations. Use it on every answer that is not a refusal. It scans for [n], reports numbers outside 1..N and flags answers with no marker, treating the exact refusal sentence as valid. It guards against invented or missing citations. Adapt it by returning the sorted lists of cited and unused sources for logging (Topic 8.3).
5
Recipe: check the numbers. Use it as the cheapest grounding test. It lists numbers in the answer that appear in no source. It guards against changed prices, dates and limits. Adapt it by normalising formats like $25 and 25 dollars, and by checking ids and names the same way (Topic 8.7).
6
Where the real call plugs in. Everything above is plain code around one call. The polyShow block below shows an OpenAI-compatible chat request in each language: the sources and question go in, the answer text comes out, and then the verifier runs. The rest of the system does not change when the provider does.

Explain it without notes

Which recipe protects against which failure?

Why does the gate run before the model call?

How would you test these recipes?

What is the minimum you would log for every answer?

Practice

Write should_answer(scores, threshold) returning True only if the list is non-empty and its maximum is at least the threshold. Print the result for [], [0.38] and [0.47, 0.1] with threshold 0.40.

Write verify(answer, n_sources) returning a list of problems (an out-of-range marker gives citation N is not a source, no marker gives no citation). Print the problems for "Cost 25 [1] [2]" with 1 source and for "Cost 25" with 1 source.

Trade-offs

↔
Small recipes in your own code: easy to test and tune, and a little more code to maintain than calling a framework.
↔
Strict verification: fewer bad answers, more retries and refusals.
↔
Checking every answer: complete coverage, a few microseconds per answer; sampling saves nothing here, but it does for judge-model checks.

Done when you can

Before you ship: the prompt layout lives in one function and is tested.
Before you ship: sources are packed under a token budget that includes overhead.
Before you ship: the refusal threshold was measured on answerable and unanswerable questions and is stored with the embedder's name.
Before you ship: every non-refusal answer has its citations verified and its numbers checked.
Before you ship: superseded sources are labelled, filtered or surfaced as a conflict.
Before you ship: refusals, gate scores, citation problems and tokens used are logged.

Command Palette

Words you'll meet

Step by step

01Step 1: decide with the gate

02Step 2: pack and render

03Step 3: the model call

04Step 4: verify before showing

05Step 5: test the recipes with fixed inputs

06Step 6: log the outcomes

Try it yourself

Turn the demo into tests

Swap the token rule

Add a recipe

Code & diagrams

Break it on purpose

Allow one source too many

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can