Topic 7.9
Playbook: Query Recipes
In one line
Recipes to copy: a standalone-question prompt with a reply guard, a multi-query fuser, a router with a safe default, and a checklist with numbers for deciding whether an extra model call earns its place.
Think of it like this
A mechanic's drawer of tools. The tools are small and each does one job. The skill is knowing which one to reach for, and not using the big wrench when a screwdriver will do.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Recipe
- A small piece of code and a rule of thumb you can copy into any project.
- Timeout
- The longest a step may take before the system moves on without it.
- Added latency
- Extra waiting time the user feels from an added step.
- Threshold
- A number set in advance that a measurement must beat.
- Fallback
- What to do when a step fails: use the previous step's result.
Step by step
01Step 1: build the prompt from the last turns
Take the last two exchanges, cut each message, and add the latest message at the end.
02Step 2: guard the reply
Accept one short line, otherwise use the original message. The three outputs in the example show an accepted reply and two rejected ones.
03Step 3: fuse several lists
Add reciprocal ranks and sort. The example fuses three small lists and keeps the top two.
04Step 4: route with a default
Rules in a list, highest score above its minimum wins, anything else goes to the default.
05Step 5: decide with numbers
For each model-backed step, compare the gain in points against the added time and cost with the threshold you chose before looking at the result.
06Step 6: plug in the real call
Replace the scripted model with a real call: same input, same output, plus a timeout and the guard. Nothing else changes.
Try it yourself
- 1
Test the guard
Add a fourth reply to the first example: a string of 25 words. Predict whether the guard accepts it, then run.
- 2
Change the history limit
Set
max_turnsto 1. Which messages does the prompt keep? What kind of follow-up would now break? - 3
Tune the decision
Change
MAX_MSto 1000 in the second example. Which decision flips? Is that a good reason to flip it?
Code & diagrams
Recipe 1 (prompt and guard), recipe 2 (fuser), recipe 3 (router). The prompt keeps only the last four messages, each cut to 40 characters.
Expected output
Rewrite the last message as a standalone question. Reply with the question only. Conversation: User: And helmets? | Assistant: A helmet is included free with every ren | User: Can I cancel? | Assistant: Yes, up to 2 hours before the start. | Last message: and later?
Can I cancel less than 2 hours before the start?
and later?
and later?
D5 0.0328, D10 0.0164
Where is my booking receipt? orders
thanks! none
Do shops open on holidays? docsThe hit rates, latencies and tokens are made up to show the calculation. The price is an example: use your provider's. Measure your own numbers.
Expected output
rewrite with a model +9 points, +450 ms, 0.024 dollars per 1000 queries -> keep
standalone question +33 points, +450 ms, 0.040 dollars per 1000 queries -> keep
three phrasings fused +1 points, +900 ms, 0.080 dollars per 1000 queries -> dropNot run here: it needs a provider, a key and the network. The helpers standalone_prompt and accept_rewrite are recipe 1 and the guard from the examples above. Checked against Groq's docs: base URL, model id openai/gpt-oss-20b and max_completion_tokens (max_tokens is deprecated there). The OpenAI-compatible chat endpoint is shown with Groq's base URL; other providers use their own. Reasoning models spend hidden tokens first, so leave room in the output limit. Real Java projects use Jackson (or Gson) for JSON.
# pip install openai
# The recipes above stay as they are. Only the model call is real.
import os
from openai import OpenAI
client = OpenAI(api_key=os.environ["GROQ_API_KEY"], base_url="https://api.groq.com/openai/v1")
MODEL = "openai/gpt-oss-20b" # a small fast model; check your provider for its current list
def standalone_question(history, message):
if not history:
return message # nothing to resolve, so no call
try:
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": standalone_prompt(history, message)}],
temperature=0,
max_completion_tokens=400, # a reasoning model spends hidden tokens first: leave room
timeout=5, # seconds: a slow rewrite is worse than none
)
return accept_rewrite(response.choices[0].message.content or "", message)
except Exception:
return message # fall back to the original on any failure
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust any reply
In the Python tab, make the guard accept any non-empty reply: change the ok = ... line in accept_rewrite to ok = bool(text). Run and read the lines after the prompt.
Myth vs fact
Myth
A query layer is a one-time build.
Fact
Models, prompts and retrievers change. Re-measure each step when they do.
Myth
If a step helps on average, keep it.
Fact
Compare the gain to its cost in time and money, against a threshold set beforehand.
Myth
The guard is optional if the model is good.
Fact
Even good models sometimes answer instead of rewriting. A cheap check prevents an answer becoming a query.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Treat every model-backed step as optional at run time: timeout, fallback, and a circuit breaker that turns the step off when its error rate rises (see the LLM Engineering course on retries).
- ▸
Cache by normalised input. Rewrites, filters and routes are deterministic at temperature 0, so equal inputs can share a result.
- ▸
Keep the thresholds in configuration. The decision rule is a business choice (how much latency a point of hit rate is worth), and it will change.
Remember this
- 1
Recipe: standalone-question prompt and guard. When to use it: chat products where the user writes follow-ups. What it guards against: the retriever receiving "how much is it?". Keep the last two exchanges, cut each message to 40 characters (the example), ask for "the question only", then accept the reply only if it is non-empty, one line and at most 20 words. Adapt: change the history limit and the word limit, add the tenant or language to the instruction.
- 2
Recipe: multi-query fuser. When to use it: compound questions, or a retriever that is sensitive to wording. What it guards against: one phrasing missing the right passage. Run each query for its top 3 to 5, add
1 / (60 + rank)per passage, sort, and take the top N. Adapt: weight a query by repeating its list, or add the original question as one of the lists so a bad rewrite cannot hide it. - 3
Recipe: router with a safe default. When to use it: more than one source, or questions that need no source. What it guards against: sending a question to a place that cannot answer it. Each route is
(name, words, minimum matches), the highest score above its minimum wins, ties and weak scores go to the default. Adapt: add exact-shape rules before the scoring, and log the route with a reason string. - 4
Recipe: is the extra call worth it? When to use it: before adding any model-backed step, and again when the model, prompt or retriever changes. What it guards against: a pipeline that grows steps nobody measured. Measure the hit rate before and after on your own questions, the added milliseconds and the added tokens, and compare with a threshold you set in advance. In the example, with made-up numbers, rewriting gains 9 points for 450 ms and is kept. The standalone question gains 33 points and is kept. Three fused phrasings gain 1 point for 900 ms and are dropped.
- 5
Where the real call plugs in. All recipes use a scripted model so they run in the browser. In production,
standalone_questioncalls a model through an OpenAI-compatible endpoint with temperature 0, a small output limit and a timeout, then passes the reply through the guard. The code is shown below and is not run here. - 6
Before you ship a query layer. A short checklist: raw and rewritten queries are logged; every model-backed step has a timeout and a fallback to the previous step; each step has a switch and a measured gain; the router has a default and a test set; filters are validated and security filters come from the user's identity. The checklist at the end of this topic is the same list in short form.
Explain it without notes
What does the guard on a rewrite do, and which checks does it make?
How do you decide whether a multi-query step is worth it?
What belongs in the log of a query layer?
Why should a router have a default, and what should it be?
Practice
Write decide(gain, ms, min_gain, max_ms) returning keep or drop by the rule in the example. Print the decision for (9, 450), (33, 450) and (1, 900) with min_gain = 5 and max_ms = 600.
Write fuse_top(rankings, n) returning the top n ids by reciprocal rank fusion, joined with commas. Print it for [["a", "b"], ["c", "a"]] with n = 2.
Trade-offs
- ↔
A guard with strict checks: fewer bad queries, more fallbacks to the original.
- ↔
A decision threshold: simple and repeatable, but it sets a price on hit rate that you have to defend.
- ↔
Caching rewrites: saves calls and time, but needs invalidation when the prompt or model changes.
Done when you can
Before you ship: raw and produced queries are logged with the route and filter.
Before you ship: every model-backed step has a timeout and a fallback.
Before you ship: every step has a switch, a measured gain and a threshold.
Before you ship: the router has a default and a labelled test set.
Before you ship: filters are validated, and access rules come from the user's identity.