PHASE 14Advanced ~27 min· topic 7 of 7

Topic 14.7

Playbook: The Whole Course on One Page

In one line

Every recipe from the course, one per phase, with the topic where you built it, a symptom lookup, and four runnable recipes that are used together most often: typed calls, resilient calls, a safe tool runner and an evaluation gate.

Think of it like this

The laminated card next to a pilot's seat. The pilot has studied for years, yet the card lists the steps for the moments that matter, in order, so nothing is forgotten. This page is that card for LLM engineering: one line per phase, the recipe you built there, and where to find it again in a minute.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Recipe: A small, tested piece of code and a rule for when to use it, which you can copy and adapt.
Symptom lookup: A list that goes from what you see going wrong to the topic that fixes it.
Gate: An automatic check that stops a change from shipping when it fails.
No-regression rule: A rule that a new version must not score worse than the current one by more than a small margin.
Allow-list: A list of the only things permitted. Anything not on it is refused.

Step by step

01Step 1: see how the recipes line up on one request

A single request passes through most of the recipes in a fixed order. The diagram names them. Keep this order in your head when you read a codebase: it tells you where a missing guard would sit.

Step 1: see how the recipes line up on one requestdiagram

Rendering diagram…

02Step 2: the typed call

Run the first recipe. Three tickets: one works at once, one needs a repair after a label outside the allowed list, and one ends in the fallback after three bad replies. The error list shows what a real client would send back to the model.

03Step 3: the resilient call

The second recipe fails the primary model four times, retries with jitter, falls back to a backup, caches the answer and meters the cost. Read the clock and cost on each line: the first question costs 60 micro-dollars on the backup, the second 175 on the primary, and the repeat costs nothing.

04Step 4: the safe tool runner

The third recipe plays a scripted model that asks for a real tool, an invented tool, a wrong argument type and an unknown city. Each failure becomes an error message for the model, never a crash and never an action outside the allow-list.

05Step 5: the evaluation gate

The fourth recipe scores two prompt versions on five golden cases and blocks the second because it falls under 90%. Notice that whitespace and case differences are tolerated, while a different label counts as wrong.

06Step 6: plug in the real model

Every recipe uses a fake model. The last code block shows one real call_model function over plain HTTP with a timeout, error classification and the finish_reason check. Swap it in behind the same interface and nothing else changes.

Try it yourself

1
Add a ticket to the typed call
Add t4 to the script with two replies, {"label": "Billing"} (capital B) and then {"label": "billing"}. What happens on the first attempt? Decide whether the validator should lower-case the label, and change it.
2
Make the primary model fail longer
In the resilient call, change <= 4 to <= 7. Which model answers each question now, and what happens to the cost?
3
Move the gate
Set MIN_PERCENT to 70 in the evaluation gate and run it. Does v2 pass? Decide which bar your own product would need, and what evidence you would collect to justify it.

Code & diagrams

Recipe: the typed call (parse, validate, repair, fallback) New tab

The fake model replies from a script. A real client would send each error back as part of the next request.

Expected output

t1: billing after 1 attempt(s)
t2: shipping after 2 attempt(s)
    error: label must be one of billing, other, shipping
t3: other after 3 attempt(s) (fallback)
    error: no JSON object found
    error: label must be one of billing, other, shipping
    error: no JSON object found

Recipe: the resilient call (jitter, fallback, cache, cost) New tab

Seeded jitter and a simulated clock. Prices are example values in micro-dollars per million tokens.

Expected output

q1: answer to q1 [backup] (clock 114 ms, cost so far 60 micro-dollars)
q2: answer to q2 [primary] (clock 139 ms, cost so far 235 micro-dollars)
q1: answer to q1 [cache] (clock 139 ms, cost so far 235 micro-dollars)

Recipe: the safe tool runner New tab

Only tools on the allow-list can run. Every failure is returned to the model as text.

Expected output

step 1: get_weather({"city": "Pune"}) -> 28 C in Pune
step 2: delete_everything({}) -> error: unknown tool delete_everything
step 3: get_weather({"city": 5}) -> error: city must be a string
step 4: get_weather({"city": "Mars"}) -> error: no data for Mars
final: Pune is 28 C. I have no data for Mars.

Recipe: the evaluation gate New tab

An absolute bar of 90% and a no-drop rule of 5 points against the version in production.

Expected output

v1: 100%   v1: 100%   wrong: none
gate v1 -> v1: PASS
v1: 100%   v2: 80%   wrong: c3
gate v1 -> v2: BLOCK: 80% is under the 90% bar

Where the real model plugs in: one call_model functionpython

Not run here: needs your own GROQ_API_KEY, the network and httpx (Python) or Jackson (Java). It uses plain HTTP so it never depends on an SDK version.

# pip install httpx. Replace the fake model in the recipes with this one function; nothing else changes.
import os

import httpx


class ModelError(Exception):
    def __init__(self, status, retryable):
        super().__init__(f"HTTP {status}")
        self.status, self.retryable = status, retryable


def call_model(system, user, model="openai/gpt-oss-20b"):
    try:
        response = httpx.post(
            "https://api.groq.com/openai/v1/chat/completions",
            headers={"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]},  # the key stays in the environment (Topic 2.4)
            json={
                "model": model,
                "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
                "max_completion_tokens": 800,  # reasoning tokens count against this limit too (Topic 1.8)
                "reasoning_effort": "low",     # this model reasons before it answers
                "temperature": 0,
            },
            timeout=30,  # always a timeout (Topic 2.7)
        )
    except httpx.TimeoutException:
        raise ModelError(408, retryable=True)
    if response.status_code != 200:
        # 408, 429 and 5xx are worth retrying (Topic 2.6); the rest are your bug.
        # A 429 that says your quota or spend cap is used up does not clear on retry: read the error body first.
        raise ModelError(response.status_code, retryable=response.status_code in (408, 429) or response.status_code >= 500)
    body = response.json()
    choice = body["choices"][0]
    if choice["finish_reason"] == "length":
        raise ModelError(200, retryable=False)  # cut off: raise max_tokens, do not parse half an answer
    return choice["message"]["content"], body["usage"]  # log the usage with the request id (Phase 11)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the JSON without validating it

In parse_and_validate, replace if data.get("label") not in LABELS: with if False: (Python), so no label is checked.

terminal

$ Run

── what you'll see ──

t1: billing after 1 attempt(s)

t2: refunds after 1 attempt(s)

t3: 5 after 2 attempt(s)

error: no JSON object found

Myth vs fact

Myth

A framework removes the need for these recipes.

Fact

Frameworks package the same ideas. If you cannot write the core, you cannot debug it when it misbehaves.

Myth

You need all recipes from day one.

Fact

Start with the first-feature checklist, the production client and the typed call. Add the others when a test, a bill or an incident shows the need.

Myth

Recipes are only for big teams.

Fact

A solo project benefits most: a small gate and a cost meter catch problems you would otherwise find in production.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Order matters: validate input, check the cache, retrieve, call with retry and fallback, run tools, validate output, log. A cache placed after the model call saves nothing, and a guard placed after the action protects nothing.
▸
Version everything that changes behaviour (prompt, model, retrieval settings, guard rules) and log the versions with each request, so any answer can be reproduced and any regression traced.
▸
Keep each recipe's tests with the recipe. When you copy a recipe into a project, copy its tests too; they are the spec of what it guards against.

Remember this

1
Recipe: your first LLM feature (Phase 0, Topic 0.7). Use it before you write any code. Check that an LLM is the right tool, fix the input and output shape, start with a fake model, set limits, keep the key on the server, log every call. It guards against the demo that fails in production. Adapt the checklist to your team's review process.
2
Recipe: budget tokens and pick temperatures (Phase 1, Topic 1.10). Count tokens with the real tokenizer or the usage field, reserve room for the reply, use a low temperature for extraction and code, higher for creative text. It guards against truncated replies and wild variation. Adapt the numbers per model.
3
Recipe: a production model client (Phase 2, Topic 2.9). One class behind an interface with a timeout, error classification, the key from the environment and usage logging. It guards against hangs, leaked keys and untestable code. Adapt it by adding providers behind the same interface.
4
Recipe: prompt patterns (Phase 3, Topic 3.9). Versioned templates for classify, extract, summarise, rewrite and answer-from-context, with data kept apart from instructions. It guards against vague prompts and injection through string mixing. Adapt by adding a regression case for every bug you fix.
5
Recipe: the typed call (Phase 4, Topic 4.8). Call, parse, validate, repair with the error message, fall back. Runnable below. It guards against broken JSON and invented values. Adapt the schema, the attempt limit and the fallback to your product.
6
Recipe: streaming (Phase 5, Topic 5.8). An SSE parser, a stream assembler and a consumer that cancels cleanly (the parser is in Topic 14.4). It guards against split chunks, lost usage and paying for tokens nobody reads. Adapt it to your provider's fields.
7
Recipe: resilient, affordable calls (Phase 6, Topic 6.11). Limiter, retry with full jitter, fallback model, cache and cost meter. Runnable below. It guards against rate limits, outages and surprise bills. Adapt the ceilings, the model list and the prices.
8
Recipe: the safe tool runner (Phase 7, Topic 7.9). An allow-list of tools, argument validation, timeouts, error feedback and a step limit. Runnable below. It guards against invented tools and bad arguments. Adapt by marking each tool read-only or side-effecting.
9
Recipe: a minimal RAG service (Phase 8, Topic 8.8). Chunk, index, retrieve, cite, refuse. The capstone skeleton in Topic 14.5 shows it wired in. It guards against made-up answers. Adapt chunk sizes and the refusal threshold with a golden set.
10
Recipe: the evaluation gate (Phase 9, Topic 9.8). Golden set, scorers, report, and a threshold plus a no-regression rule in CI. Runnable below. It guards against silent quality drops. Adapt the set to real failures.
11
Recipe: guards and the security checklist (Phase 10, Topic 10.9). Input guard, output guard, redactor and a tool allow-list with least privilege. It guards against injection, leaks and unsafe output. Adapt by adding your own misuse cases to the test set.
12
Recipe: an instrumented client (Phase 11, Topic 11.8). Redacted structured logs, a span and metrics around every call. It guards against not knowing what happened. Adapt the metric names to your dashboards.
13
Recipe: the service skeleton (Phase 12, Topic 12.9). An endpoint with limits, streaming, storage and observability, config and flags. It guards against one-off scripts that cannot be operated. Adapt storage and deployment to your platform.
14
Recipe: the model scorecard (Phase 13, Topic 13.8). Weighted criteria, your own evals, a decision record. It guards against choosing from a leaderboard. Adapt the weights to what your product values.
15
Symptom lookup. Reply cut off or empty: check finish_reason and max_tokens, and remember that a reasoning model spends its hidden reasoning tokens from the same limit (Topics 1.8, 2.3). Broken JSON: the typed call (4.8). 429 or timeouts: Topics 2.6, 6.8; a 429 from an exhausted quota or spend cap does not clear on retry, so check billing and the error body instead of backing off. Slow first word: streaming (5.1). Bill too high: Topics 6.1 to 6.6. Wrong facts: grounding and refusal (8.6). Quality dropped after a change: the evaluation gate (9.8). Strange behaviour from user text: Topic 10.2. Cannot tell what happened: Topics 11.2 to 11.4.

Explain it without notes

If you had one day to make a prototype production-ready, which recipes would you add first, and in what order?

How do you decide whether a new recipe or guard is worth adding?

A quality drop is reported a week after a prompt change. Which recipes would have caught it, and how?

Practice

Write pick_label(raw, allowed) that returns the one allowed label found (ignoring case) in the model's raw reply, or other when none or more than one is found. Print results for Billing, I think shipping or billing and no idea with the allowed labels billing and shipping.

Write tally(events) that counts outcomes such as ok, retry and fallback in a list, and print the counts sorted by name.

Trade-offs

↔
More recipes mean more safety and more code to maintain; adopt them in the order that failures actually cost you.
↔
Copying a recipe is fast, but you own it afterwards; keep its tests with it and note where it came from.
↔
A strict gate protects quality but can block good changes on noise; use a margin and enough cases, and review blocked changes.

Done when you can

Before you ship: the model sits behind an interface with a timeout, error classes and a key in the environment.
Before you ship: every output is validated, with a repair limit and a safe fallback.
Before you ship: retries use jitter and a cap, never retry a quota or spend-cap 429, and there is a limiter, a fallback model and a cost meter.
Before you ship: tools are allow-listed, validated and have a step limit, and side effects need approval.
Before you ship: answers cite retrieved sources and refuse when there is no evidence.
Before you ship: a golden set runs in CI with a threshold and a no-regression rule.
Before you ship: logs are redacted, metrics and alerts exist, and a kill switch is ready.
Before you ship: you can name the prompt, model and settings version of any answer.

Command Palette

Words you'll meet

Step by step

01Step 1: see how the recipes line up on one request

02Step 2: the typed call

03Step 3: the resilient call

04Step 4: the safe tool runner

05Step 5: the evaluation gate

06Step 6: plug in the real model

Try it yourself

Add a ticket to the typed call

Make the primary model fail longer

Move the gate

Code & diagrams

Break it on purpose

Trust the JSON without validating it

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can