Retrieval in a Conversation · RAG and Vector Search

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Standalone question: A question rewritten so it can be understood without the conversation.
Follow-up: A message that depends on earlier messages, such as "and on holidays?".
History: The earlier messages of the conversation, from the user and the assistant.
Coreference: A word like "it" or "the second one" that refers to something said earlier.
Small talk: Greetings and thanks that need no documents.
Retrieve decision: The choice, made per message, to search or not.

Step by step

01Step 1: see a follow-up fail

Search "How much is it?" alone and you get no result at all. "do I pay for the other one?" retrieves the helmet passage because of the word "one". The retriever is doing its job on the text it was given.

02Step 2: build the rewrite prompt

A short instruction, the conversation in order, then the last message. Ask for the question only.

standalone.pywhole filepython

def build_prompt(history, message):
    turns = " | ".join(role + ": " + text for role, text in history)
    return ("Rewrite the last message as a standalone question using the conversation. "
            "Reply with the question only. Conversation: " + turns + " | Last message: " + message)

03Step 3: skip the call when there is nothing to resolve

No history means the message is already the question. Return it unchanged and make no call.

04Step 4: compare raw and standalone searches

The example prints both. The standalone question reaches the right passage with a higher score every time.

Step 4: compare raw and standalone searchesdiagram

Rendering diagram…

05Step 5: decide whether to retrieve

Small talk and requests about the last answer need no search. Everything else retrieves. The default is to retrieve.

06Step 6: try a real model

The lab sends a conversation to a real model and asks for the standalone question. The first search lab shows why: the follow-up "What about the second one?" has no signal, and the standalone question has plenty.

Try it yourself

1
Add a new conversation
Add a history about the helmet passage and the follow-up "and for kids?". Add a scripted reply for it. What does the raw search find, and what does the standalone one?
2
Trim the history
In the prompt builder, keep only the last two messages. Is the prompt still enough for the scripted examples? When would it not be?
3
Test the retrieve rule
Add "Can you make the refund rule shorter?" to the second example. What does the rule decide? Is that right?
4
Use the lab
In the real-model lab change the last message to "What about the first one?" and send. Does the model resolve it to the late fee?

Code & diagrams

Standalone questions from history New tab

The model is scripted, one reply per follow-up. A real one reads the history and writes the reply itself.

Sign in to run this example in your browser.

Expected output

message:     and on holidays?
standalone:  What time do shops close on public holidays? (model called)
raw search:  D2 (1)   standalone search: D2 (4)
message:     How much is it?
standalone:  How much is the monthly pass? (model called)
raw search:  none   standalone search: D8 (2)
message:     Do I pay for the other one?
standalone:  Do I pay for another bike after a flat tyre? (model called)
raw search:  D4 (1)   standalone search: D10 (4)
message:     What does the monthly pass cost?
standalone:  What does the monthly pass cost? (no history, no call)
raw search:  D8 (2)   standalone search: D8 (2)

When not to retrieve New tab

Rules for "do not retrieve" are narrow on purpose. Anything unrecognised retrieves.

Sign in to run this example in your browser.

Expected output

hello                             no retrieval (small talk)
Thanks a lot!                     no retrieval (small talk)
Can you make that shorter?        no retrieval (about the last answer)
hi, how do I reset my password?   retrieve
What does the monthly pass cost?  retrieve

A follow-up that points at nothingreal embeddings

Try: Earlier the assistant said: "Late returns cost 2 dollars per 15 minutes, and damage must be reported in the app." You asked "What about the second one?". Press Search: the best score is tiny and the top passage is arbitrary. Then replace the question with the standalone question you would have written (what do you do if the bike is damaged?) and compare the scores.

QuestionPassages (12)

1

2

3

4

5

6

7

8

9

10

11

12

Rank byTop k (retrieved)

Embedding modelall-MiniLM-L6-v2384 numbers per text. Runs in your browser: the first Search downloads about 23 MB once, and your text never leaves this page.

Goals

The damage passage is the best match
The damage passage scores at least 0.50

Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.

A real model writes the standalone questionreal model

Try: The conversation is in the message. Send it and read the reply: did the model resolve "the second one" to the damage rule? Then change the system message (for example, ask for a keyword query instead of a question) and compare.

System

user

Modelmid-size, also reads imagesTemperature · 0.0Max tokens (1–1024) Stream (SSE) JSON mode

Goals

The reply is a short question (25 words or fewer)
It resolved "the second one" to damage
It does not answer, only rewrites

Sign in to send this request to a real model (free).

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Treat "a" as a content word

In the Python tab of the second example, stop allowing ordinary function words in small talk: change all(word in SMALL_TALK or word in STOP for word in words) to all(word in SMALL_TALK for word in words). Run.

terminal

$ Run

── what you'll see ──

hello no retrieval (small talk)

Thanks a lot! retrieve

Can you make that shorter? no retrieval (about the last answer)

Myth vs fact

Myth

The retriever can read the chat.

Fact

A retriever sees only the text you send it. If the question points back, the retriever cannot follow.

Myth

Just send the whole history to the retriever.

Fact

Long histories mix topics and drown the question. Resolve it into one clear question first.

Myth

Never skip retrieval, to be safe.

Fact

Searching for "thanks a lot" wastes time and can put an unrelated passage in the prompt. Skip it for clear small talk only.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Standalone rewriting adds one model call per turn. Use the cheap rule (history exists and the message is short or contains a pointer word) to avoid it on obviously self-contained turns.
▸
Evaluate on conversations, not single questions. A golden set for chat needs history plus the follow-up plus the expected passage (Topic 9.2).
▸
When the user switches topic, the old history misleads the rewrite. Cap the history, and tell the model to ignore it when the last message is already clear.

Remember this

1
Why this exists. The retriever has no memory. It sees only the text you send. In a chat the user writes the way people talk: "and on holidays?", "how much is it?", "do I pay for the other one?". Searched as written, these retrieve noise: in the example, "do I pay for the other one?" retrieves D4 (the helmet passage) because of the word "one". It is not the conversation's fault, it is the retriever being handed half a sentence.
2
The standalone question. Before retrieval, rewrite the last message into one that makes sense alone, using the history: "How much is it?" after a chat about the monthly pass becomes "How much is the monthly pass?". The search then scores D8 with 2 shared words instead of finding nothing. "and on holidays?" becomes "What time do shops close on public holidays?" and D2's score goes from 1 to 4. "do I pay for the other one?" after a flat tyre becomes "Do I pay for another bike after a flat tyre?", and the right passage D10 scores 4.
3
Which part of the history to send. The whole chat is expensive and noisy. The last one to three exchanges are usually enough, each cut to a limit, so the prompt stays small and the model is less likely to follow an old topic. Topic 7.9 has this as a recipe. Include both the user's and the assistant's messages: "the second one" usually refers to something the assistant listed.
4
No history, no call. On the first message there is nothing to resolve, so skip the model call. The example's last case sends a first-message question straight to the retriever with no model call. The same applies to messages that are clearly self-contained. A cheap rule (a history exists and the message is short or contains a pointer word) avoids paying for the call every turn.
5
Keep the raw message too. The standalone question is the model's reading of the conversation. If the model resolves "it" wrongly, the search is wrong and nobody sees why. Log the standalone question with the request. In a high-stakes system, show it to the user ("Searching for: How much is the monthly pass?") so they can correct it.
6
When not to retrieve at all. "Hello", "thanks a lot" and "can you make that shorter?" do not need the handbook. The first two are small talk. The third is about the previous answer and needs the conversation, not new documents. A small rule can skip retrieval in those cases, and the safe default for any message it does not recognise is to retrieve: a wasted search costs a few milliseconds, a missed search costs a wrong answer.
7
The rule has limits. "Can you make the refund rule shorter?" contains shorter but also asks about a topic, and a crude rule would skip retrieval for it. Rules for "do not retrieve" must be narrow, with the default of retrieving. The lab below uses a real model for the rewrite, where you can see how a real one handles a vague follow-up.

Explain it without notes

01

Why can't a retriever answer "how much is it?"?

02

What goes into a standalone-question prompt, and what do you leave out?

03

Why not make a model call on every turn?

04

Why must the rule for skipping retrieval be narrow?

Practice

01

Write needs_rewrite(history, message) that returns True when the history is non-empty and the message has four words or fewer, or contains one of it, that, one. Print yes or no for three cases.

02

Write last_turns(history, n, limit) returning the last n messages, each cut to limit characters. Print the result for six short messages with n = 4 and limit = 10, joined with " | ".

Trade-offs

↔
A rewrite call every turn: robust, but costly. A rule that skips obvious cases: cheaper, with some risk of missing a case.
↔
Sending more history: more context for the rewrite, more noise and cost.
↔
Skipping retrieval for small talk: faster and cheaper, safe only with a narrow, tested rule.

Done when you can

Done when you can build a standalone-question prompt from history and a follow-up.
Done when you can say why no history means no model call.
Done when you can write a narrow rule for skipping retrieval, with retrieve as the default.
Done when you can show a follow-up failing raw and working as a standalone question.

Command Palette

Words you'll meet

Step by step

01Step 1: see a follow-up fail

02Step 2: build the rewrite prompt

03Step 3: skip the call when there is nothing to resolve

04Step 4: compare raw and standalone searches

05Step 5: decide whether to retrieve

06Step 6: try a real model

Try it yourself

Add a new conversation

Trim the history

Test the retrieve rule

Use the lab

Code & diagrams

Break it on purpose

Treat "a" as a content word

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can