Topic 11.11
Playbook: Production RAG Recipes
In one line
Four recipes to copy into any RAG project: a traced request, cache keys with invalidation, a latency budget table, and a migration runbook for changing the model. Plus a launch checklist for the production side.
Think of it like this
A pilot's checklist. Each item looks obvious, and each exists because someone once skipped it. These recipes are the same: small, boring and written down, so the production side of RAG does not depend on anyone remembering.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Recipe
- A small, reusable piece of code or procedure with a clear purpose.
- Checklist
- A short list of things to verify before a step, so none is forgotten.
- Span
- One timed stage in a trace.
- TTL
- Time to live: how long a cache entry may be used.
- Dual write
- Writing each change to both the old and the new index during a migration.
- Owner
- The person or team responsible for acting on an alert.
Step by step
01Step 1: put the tracer on the query path
Create one Tracer per request with its id. Call stage(name, ms, attrs) around each stage. Write line() to your logs when the request ends.
02Step 2: put the cache behind a key function
Never build keys inline. One function, Cache.key(...), takes every ingredient, so adding one (a tenant, a prompt version) is a change in one place.
03Step 3: register dependencies when you store
cache.put(key, value, doc_ids) takes the documents the answer used. Ingestion calls cache.invalidate(doc_id) for every changed document.
04Step 4: check the budget on every trace
over_budget(timings) compares each stage with its row and returns the offenders. In the example the 120 ms search is over its 80 ms row by 40.
05Step 5: keep the migration steps next to the code
Put the runbook in the repository beside the ingestion code, with the commands. Update it after every migration; a migration done once from memory is a migration done wrongly next time.
06Step 6: review the checklist before launch
Go through it with someone who did not build the system. Every unchecked item is either done, consciously accepted with an owner, or a task.
Try it yourself
- 1
Add a tenant
Put a tenant id into
Cache.key(...). Why must two tenants never share an entry, even for identical questions? - 2
Add an ingredient
Add a "retrieval settings" version to the key. Which part of the pipeline would make you bump it?
- 3
Fail the budget
Change the search time to 70. Does the budget line change? Then set the search budget to 60 and see the new message.
Code & diagrams
Production-shaped but small: the tracer uses simulated time, the cache is in memory. Swap in your tracing library and shared cache; the shapes stay.
Expected output
req=r1 embed=30 search=120 rerank=120 generate=310 total=580
budget: search over by 40
key: toy-v1|index8|p3|what does the monthly pass cost
get before edit: hit
invalidate D8 removed 1 entries
get after edit: missThis is shape, not a specific SDK: your vector store, model client and cache each have their own API. Keys come from environment variables.
# Pseudocode: names are placeholders, check your own libraries' documentation.
import os
tracer = Tracer(request_id)
cache = Cache() # in production: a shared store with a TTL (for example Redis)
def handle(question, user):
key = Cache.key(EMBED_MODEL, "index" + str(index.version), PROMPT_VERSION, user.tenant, normalize(question))
cached = cache.get(key)
if cached:
return cached
tracer.stage("embed", ...) # wrap each real call and record its time
hits = index.search(...) # your vector store client
answer = llm.generate(...) # your model client, key from os.environ
cache.put(key, answer, [hit.doc_id for hit in hits])
return answerBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Invalidate nothing
In the Python tab, in Cache.invalidate, change keys = self.deps.pop(doc_id, set()) to keys = set() and run.
Myth vs fact
Myth
Recipes like these are over-engineering for a small service.
Fact
They are small: a tracer, a key function and a table. They cost an afternoon and prevent the three most common production problems (unknown slow stage, stale answers, risky model change).
Myth
A migration runbook is only needed for model changes.
Fact
Any change that alters vectors or chunk ids (the chunker, preprocessing, the dimension) needs the same steps.
Myth
The launch checklist is the quality bar.
Fact
It covers the operating side. Quality (Phase 9) and security (Phase 10) have their own checklists, which Phase 13 combines.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Make the recipes one module the whole team imports, so every service traces and keys caches the same way.
- ▸
Keep a change log of index versions: version, model, chunker, date and why. It turns "it got worse last week" into a lookup.
- ▸
Test the rollback as often as the deploy. A rollback that has never been run is a guess.
- ▸
Automate the cut-over rule (hit rate not lower, no regressions) as a CI job on the golden set (Topic 9.7), so a person decides only edge cases.
Remember this
- 1
Recipe: a traced request. When to use it: every query path. What it guards against: not knowing which stage was slow or which index version answered. How to adapt it: give each stage a span (name, start, end, attributes), pass one request id through everything, and write one summary line per request (
req=r1 embed=30 search=120 ... total=580). Add the index version and model name as attributes. Never log passage text by default. - 2
Recipe: cache keys with invalidation. When to use it: any cache between the user and the model. What it guards against: stale or foreign answers. How to adapt it: build the key from every ingredient (
model|index version|prompt version|normalised question, plus tenant or permission scope when answers differ), register each entry against the documents it used, and delete those entries when a document changes. Keep a TTL as a second line of defence. - 3
Recipe: a latency budget table. When to use it: as soon as you have users. What it guards against: slow drift and vague complaints. How to adapt it: one row per stage with a p95 budget; compute p50 and p95 from real traces; alert when a stage exceeds its row; check the whole request against its own budget because percentiles do not add. Pass a deadline down so late stages can skip themselves.
- 4
Recipe: a migration runbook. When to use it: changing the embedding model, the chunker or any step that changes vectors or chunk ids. What it guards against: mixed vectors, a stale new index, and a switch with no way back. How to adapt it, in order: tag the old index; build the new index beside it by backfilling in batches; dual-write changes; run the test questions against both; cut over only if nothing regressed (a canary share first); keep the old index and the dual write until the new one has had a calm period; then delete.
- 5
Recipe: an incident runbook table. When to use it: before launch. What it guards against: improvising at 3 in the morning. How to adapt it: one row per known failure (stale, shrunken index, leak, cost spike, plus your own) with the signal, the first check, the first safe action and the owner; order alerts by harm; drill it once a quarter (Topic 11.9).
- 6
The launch checklist (production side). Both paths deployed separately; the index tagged with model and version; freshness allowance written and lag alerted; caches keyed completely and invalidated by document; budgets per stage and for the request; cost per answer computed and watched; logs, spans and the five core metrics in place; shards and replicas sized with headroom (only if needed); a tested rollback for index and model; runbooks for four incidents; a canary question in the health check. Phase 13's launch checklist adds quality and security.
Explain it without notes
What goes into a RAG cache key, and what must happen to entries when a document changes?
How would you use a latency budget table in operations?
Walk through the migration runbook for changing the embedding model.
What would you check before launching a RAG service, on the operations side?
Practice
Write cut_over_allowed(old_hits, new_hits, regressions) returning True only when new_hits is at least old_hits and the list of regressions is empty. Print yes or no for the cases (3, 5, []), (3, 5, ["Q1"]) and (5, 4, []).
Write batches(ids, size) that splits a list into consecutive lists of at most size ids, and print the lengths of the batches for 8 ids and size 3.
Trade-offs
- ↔
Shared recipes in one module: consistency, and one more dependency for every service.
- ↔
Strict launch checklists: fewer surprises, and some friction for small changes.
- ↔
Long-kept old indexes: safe rollback, ongoing cost.
Done when you can
Done when you can copy the tracer, cache and budget recipes into a project and adapt them.
Done when you can write a complete cache key for your own service.
Done when you can state your stages' budgets and measure them.
Done when you can write the migration runbook for your own stack.
Done when you can go through the production launch checklist item by item.