Topic 12.7
Retrieval as a Tool
In one line
Instead of searching once, let the model decide when to search and what to search for, look at the results, and search again. A step limit and a repeat check keep the loop from running away.
Think of it like this
A detective does not look up one thing and announce a verdict. She checks one clue, sees what it points to, asks a new question and checks again, until she knows or gives up. Fixed search is one look. Agentic retrieval is the loop.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Agentic retrieval
- A retrieval loop where the model decides the next search from what it has seen.
- Tool call
- A model's request to run one of your functions with given arguments.
- Step limit
- The most steps a loop may take. Enforced in code.
- Repeated query
- The same search asked again. A sign the loop is stuck.
- Scripted model
- A stand-in that follows a fixed list of actions, used in examples so output is exact.
- Loop budget
- A limit on steps, tokens and cost across the whole loop.
Step by step
01Step 1: a question of two parts
"What happens if I return late, and what does the monthly pass cost?" has one fact in D6 and another in D8. One search for the whole sentence matches one of them best.
02Step 2: give the model a search tool
Describe search(query) to the model. It may reply with a search request or with a final answer. The example replaces the model with a script, so you can see the decisions.
03Step 3: run the loop
At each step: if the reply is an answer, stop. If it is a search, run it and show the result. The first question takes two searches and an answer, in three steps.
04Step 4: the question with no answer
For "Do you rent scooters?" the script keeps searching for scooter hire. Each search finds nothing, and the limit stops the loop at step 3. Without the limit it would not stop.
05Step 5: add a repeat check
Remember the queries already tried. A repeat means the model is stuck and the loop should stop early, which saves the remaining steps.
06Step 6: what to return when stopped
When the loop is stopped, answer with what was found, or refuse. Never return the half-finished conversation to the user.
Try it yourself
- 1
Raise the limit
Set
MAX_STEPSto 4. What changes for the scooter question, and what does each extra step cost in a real system? - 2
Write a better script
For the scooter question write a script that searches once and then answers
answer: I could not find that in the handbook.. Compare the cost with the looping script. - 3
Make the search stricter
Change the
>= 2insearch_toolto>= 1. Which queries now find something that is not really an answer?
Code & diagrams
The model is a script so the output is exact. Real models decide differently, which is why the limit exists.
Expected output
Q: What happens if I return late, and what does the monthly pass cost?
step 1 search [bikes returned late] -> D6
step 2 search [monthly pass cost] -> D8
step 3 answer: Late returns cost 2 dollars per 15 minutes [D6]. The monthly pass costs 25 dollars [D8].
Q: Do you rent scooters?
step 1 search [scooter hire] -> nothing
step 2 search [scooter hire] -> nothing
step 3 search [scooter hire] -> nothing
stopped: step limit 3 reachedBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Remove the step limit
In the Python tab, set MAX_STEPS = 6 and extend the scooter script with three more search: scooter hire entries before the answer. This is a loose limit, standing in for no limit. Run.
Myth vs fact
Myth
A loop that lets the model search again is always more accurate.
Fact
It can fix a poor first query and it can also wander. Measure it against the fixed pipeline on your golden set.
Myth
Telling the model to stop after three searches is enough.
Fact
A prompt is a request. The limit must be enforced in code.
Myth
Agentic retrieval needs a big framework.
Fact
The core is a loop of about twenty lines. A framework helps with many tools, which is another course.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log every step: the query, the number of results and the scores. A trace of the loop is the only way to see why it took 5 steps (Topic 11.6).
- ▸
Compare cost and quality against a fixed pipeline with query rewriting and multi-query (Topic 7.4). Many questions that look like they need a loop do not.
- ▸
Cap tokens across the loop too. Each step re-sends the conversation, so cost grows faster than the number of steps: if the question and instructions take 500 tokens and each search result adds 800, the input is about 500, 1,300, 2,100 and 2,900 tokens over four calls, 6,800 in total, against 500 for one call.
Remember this
- 1
Why this exists. One search cannot answer a question made of parts ("what is the late fee, and what does the monthly pass cost?"), a question where the second search depends on the first answer, or one where the first query was poor and the result shows it. Topic 7.4 split compound questions in code before searching. Here the model decides each next step after seeing what the last one found.
- 2
Search as a tool. In tool calling (Topic 7.2 of LLM Engineering, the tool-call loop by hand) you describe a function to the model, and it replies with a request to call it with arguments. You make the call and send the result back. Here the tool is
search(query). The loop is: send the question and the tool description, if the model asks to search then run it and send the results back, repeat until the model gives a final answer. - 3
The loop must be bounded. A model can ask for the same search over and over, or keep trying phrasings when the corpus has nothing. The example has a step limit of 3 and the second question, about scooters, uses all of them and is stopped. The limit lives in your code, not in the prompt: a prompt asks, code enforces. Also stop on a repeated query, and count total tokens and money across the loop, because every step re-sends the growing conversation.
- 4
Each step must still be safe. Every result in the loop is retrieved text, so the rules of Phase 10 apply at every step: access filters on every search, results labelled as data, citations checked at the end. Calling search many times means many chances for an injected instruction to be read, so the scope of what the model can do after each step must stay as narrow as the first.
- 5
Know what it costs. Each extra step is a model call plus a search, and the context grows. A question that took 1 call and 1 search can take 4 and 3, with the latency of all of them added (Topic 11.4). Use the loop for questions that need it, and send simple ones through the fixed pipeline. A cheap router (Topic 7.6) can decide.
- 6
Where this ends. Planning across many different tools, keeping memory between sessions, and recovering from failed actions are the subject of the coming AI Agents course. This topic covers only the case where the tool is search. A fixed pipeline remains the right default: it is cheaper, faster and easier to test.
Explain it without notes
What kinds of question need more than one search?
How does the loop work, and where do the limits belong?
What new risks come from repeating searches?
When would you not use agentic retrieval?
Practice
Write parse_action(text) that splits a model reply of the form search: <query> or answer: <text> into a (kind, argument) pair, and returns ("invalid", text) for anything else. Print the result for three replies.
Write stop_reason(step, max_steps, queries, new_query) that returns "repeated query" when new_query is already in queries, "step limit" when step is at least max_steps, and None otherwise (check the repeat first). Print it for three situations.
Trade-offs
- ↔
A loop can fix poor first queries and handle chains, with more cost and latency and less predictability.
- ↔
A low step limit: cheap and safe, may stop early. A high limit: more chances, more cost.
- ↔
A framework: fast to start and heavy. A small loop: clear and easy to test.
Done when you can
Done when you can write a search loop with a step limit and a repeat check.
Done when you can say why the limit belongs in code.
Done when you can list the safety checks that must apply at every step.
Done when you can say when a fixed pipeline is the better choice.