Topic 7.2
Rewriting the Query
In one line
Rewriting turns the question the user typed into the question the retriever should see: first with plain rules (lower-case, drop greetings, fix known typos), then with a model for what rules cannot do. Keep the original, guard the model's reply, and count what the extra call costs.
Think of it like this
A receptionist hears "hey um so like where do I, you know, pay for the parking thing" and writes on the form: "Where can I pay for parking?" The visitor's words were fine for a person, but the form is what the filing clerk searches. A rewrite is that translation from spoken to searchable.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Rewrite
- Producing a better query from the user's question, by rules or by a model.
- Normalisation
- Mechanical cleaning such as lower-casing and removing punctuation.
- Guard
- A check on a model's reply before it is used, with a safe fallback.
- Fallback
- What the system does when a step fails or looks wrong. Here: search with the original question.
- Added latency
- Extra waiting time the user feels because of an additional step.
- Scripted model
- A fake that returns fixed replies so the code around it can be tested.
Step by step
01Step 1: write the rule rewrite
Tokenise, drop the filler words, fix known typos and join. Four lines of code, no model.
DROP = {"hey", "hi", "hello", "pls", "please", "thanks", "thx", "um"}
FIX = {"pasword": "password", "helmat": "helmet", "lte": "late"}
def rewrite(question):
return " ".join(FIX.get(word, word) for word in tokenize(question) if word not in DROP)02Step 2: search with both and keep the better
Run the original and the rewrite through the same retriever. Keep the rewrite unless its top score is lower. A rewrite that makes things equal costs nothing.
03Step 3: read what rules fixed and what they could not
The password question and the helmet question improve ("none" becomes D4 for the helmet one). The refund typo is fixed but still finds nothing, because the handbook says "refunded". That gap is vocabulary, not spelling, and it is what Topic 7.3 addresses.
04Step 4: ask a model for the harder cases
Rules cannot turn "hours?" into a question. A model can. Build a strict prompt and call it. The example uses a scripted model with the same input and output as a real one.
05Step 5: check the reply before trusting it
Reject empty replies and replies that are too long. Fall back to the original. The reject branch runs for the "late fee" question.
06Step 6: price the step
Sum the latency and the tokens over a batch. Four calls at 450 ms is 1800 ms of waiting, and every user pays it before any search happens. Later you will compare that to the hit-rate gain.
Try it yourself
- 1
Add a typo
Add
"refnd"to a question and run the first example. It becomesrefund. Does it find anything? Then add"refunded"to the question text itself and compare. - 2
Loosen the guard
In the second example change
MAX_WORDSto 40. What happens to the "late fee" question? Is the answer-shaped reply a good search query? - 3
Make the rewrite wrong on purpose
Change the scripted reply for
"pass price"to"How do I reset my password?". Predict the new hit, then run. Which part of the example would have to notice a wrong rewrite? - 4
Price your own provider
Replace
CALL_MS = 450with the latency you measured on your provider. How much waiting do 100 questions add?
Code & diagrams
The "kept" line says rewritten when the rewrite scores the same or better, and original when it scores lower.
Expected output
Q: Hey pls how do I reset my pasword
rewritten: how do i reset my password
original: D3 (1)
rewritten: D3 (2)
kept: rewritten
Q: are kids allowed without a helmat thx
rewritten: are kids allowed without a helmet
original: none
rewritten: D4 (1)
kept: rewritten
Q: What does the monthly pass cost?
rewritten: what does the monthly pass cost
original: D8 (2)
rewritten: D8 (2)
kept: rewritten
Q: charged if i am lte
rewritten: charged if i am late
original: D5 (1)
rewritten: D6 (2)
kept: rewritten
Q: can I get a refnd
rewritten: can i get a refund
original: none
rewritten: none
kept: rewrittenThe model is a scripted stand-in with one deliberately bad reply (the late-fee answer). The 450 ms per call is an assumed number: measure your own provider.
Expected output
Q: hours?
rewrite: What time do the shops open and close?
hit before: D1 (1) after: D2 (3)
Q: I can't get in
rewrite: How do I reset my password?
hit before: none after: D3 (2)
Q: pass price
rewrite: How much does the monthly pass cost?
hit before: D8 (1) after: D8 (2)
Q: late fee
rewrite: rejected (31 words), keeping the original
hit before: D6 (1) after: D6 (1)
extra model calls: 4, added latency: 1800 ms, words sent and received: 118Try: The ranker starts on Keyword (BM25) and the question has two typos. Press Search: no passage shares a word, so every score is 0. Switch Rank by to Vector (cosine) and search again. Then try "how do i reest my pasword" and see whether meaning still copes.
Goals
- The password-reset passage is the best match
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
An over-eager typo table
In the Python tab (first example), add one more entry to FIX: change "refnd": "refund"} to "refnd": "refund", "pass": "password"}. Run.
Myth vs fact
Myth
A rewrite can only help.
Fact
A rewrite is a guess. It can drop the one word that mattered or change the meaning. Keep the original and measure.
Myth
Rewriting always needs a model.
Fact
Lower-casing, filler removal and a typo table cover many real cases for free. Use a model for what rules cannot do.
Myth
If the model replies, the reply is a query.
Fact
Models often answer, explain or apologise. Check the shape of the reply before using it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cache rewrites keyed by the normalised question. Many questions repeat, and a cache removes the model call, the delay and the cost for them.
- ▸
Run the rewrite model at temperature 0 and with a small
max_tokens. The output is a single short sentence, and the same input should give the same query so that results can be reproduced. - ▸
Put the rewrite behind a timeout. If the model takes longer than a budget (say 800 ms), search with the original instead. A slow rewrite is worse than no rewrite.
Remember this
- 1
Why rewrite. The retriever works best on a clean, complete, self-contained question. Real input has greetings ("hey pls"), typos, shorthand and half sentences. Rewriting is the cheapest repair because it needs no change to the index.
- 2
Rules first. A rule-based rewrite lower-cases, drops words that carry no meaning (
hey,pls,thx), and fixes known typos from a small table. It is instant, free, identical every time and easy to test. In the example, "Hey pls how do I reset my pasword" becomes "how do i reset my password" and the right passage's score goes from 1 to 2. - 3
Rules have a ceiling. A table fixes only the typos someone thought of. "can I get a refnd" becomes "can i get a refund", but the handbook says "refunded", so nothing is found: fixing the spelling did not fix the vocabulary. And an over-eager rule can do damage: a typo table that rewrites
passintopasswordwould break the monthly pass question (the break-it below does exactly this). - 4
Keep the original, and compare. A rewrite is a guess about what the user meant. The safe pattern is to search with both and keep the rewritten result unless it scores clearly worse, or (Topic 7.4) to search with both and fuse the lists. The example's "kept" line applies the first rule: fall back to the original only when the rewrite lowers the top score.
- 5
A model can do what rules cannot. Given "hours?" a model can write "What time do the shops open and close?", turning one ambiguous word into a full question that shares words with the right passage. The prompt is short and strict: rewrite as one clear search question, reply with the question only. The example uses a scripted stand-in model so the mechanics run for real (Topic 7.9 shows where a real call plugs in).
- 6
Guard the reply. A model asked to rewrite will sometimes answer instead, or add "Sure! Here is the question:". The example accepts a reply only if it has between 1 and 12 words, and otherwise falls back to the original. The "late fee" case shows it: the stand-in model wrote a 31-word answer, so the guard rejected it and the original was searched. Without the guard, that answer would have been the query.
- 7
Typos: what a real embedding model does. Keyword search is blind to a misspelling, because no passage contains the token "pasword". An embedding model is less blind but not safe. In the lab below, "pasword rest" still ranks the password passage first with a cosine (the similarity score between two vectors, Topic 2.2: 1 means the same direction, 0 unrelated) of about 0.15, barely above the second passage at about 0.08. A longer misspelt question, "how do i reest my pasword", puts the flat-tyre passage first (about 0.29). Cleaning the spelling before the search is cheaper than hoping the model copes.
- 8
Count the cost. The example adds four model calls, 1800 ms (4 x 450 ms, a made-up latency) and 118 words of prompt and reply. That is per batch of four questions, but it is the real shape of the bill: every question pays the delay before retrieval even starts. Topic 7.9 turns this into a keep-or-drop decision.
Explain it without notes
What does a rule-based rewrite do, and what can it not do?
Why keep the original question when you have a rewrite?
What can go wrong when a model is asked to rewrite a query, and how do you guard against it?
How do you decide whether a model rewrite is worth its extra call?
Practice
Write rewrite(question, drop, fix) that lower-cases the question, removes any word in drop, applies fix (a dict of replacements) and joins with spaces. Print the rewrite of "Hi pls where is my refnd" for drop = {"hi", "pls"} and fix = {"refnd": "refund"}.
Write accept(reply, original, max_words) that returns the reply when it is non-empty and has at most max_words words, otherwise the original. Print the result for an empty reply, a good reply and a long reply with max_words = 6.
Trade-offs
- ↔
Rules: free, instant and repeatable, but narrow and able to break good questions if over-eager.
- ↔
Model rewrite: handles vague and messy input, but adds latency, cost and a new failure mode.
- ↔
Searching with both original and rewrite: safest, at the price of two searches.
Done when you can
Done when you can write a rule-based rewrite with a drop list and a typo table.
Done when you can explain why the original question is kept next to the rewrite.
Done when you can guard a model's reply and fall back to the original.
Done when you can total the latency and tokens an extra model call adds.