Topic 4.6
Repair and Retry Loops
In one line
When a reply fails validation, tell the model what was wrong and ask again, up to a fixed number of attempts, then fall back to a safe value. This fixes most failures at the cost of extra calls, so count the attempts and the money.
Think of it like this
A teacher marking homework in red pen and handing it back. "Question 3: you wrote 5, but the question asked for a fraction." The pupil fixes that one thing and returns it. The teacher doesn't say "wrong, try again" with no hint, and doesn't allow endless resubmissions: after the third try, the teacher writes a mark and moves on.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Repair
- Asking the model to fix a reply that failed validation, by telling it what was wrong.
- Attempt
- One call to the model within a retry loop.
- max_attempts
- The limit on how many times the loop may call the model.
- Fallback
- A safe, planned outcome used when all attempts fail.
- Feedback message
- The text sent back to the model that says what was wrong with its reply.
- Repair rate
- The share of requests that needed at least one repair. A health number for your prompt.
Step by step
01Step 1: one check function
check(text) returns either a value or a problem sentence: not JSON, wrong keys, or a plan that is not allowed. It is the validator of Topic 4.5 in small form. The loop only needs to know whether it passed and what to say if it did not.
02Step 2: the loop
ask_with_repair starts with the user's prompt. On each attempt it adds the size of the conversation to a running total, asks the model, and checks the reply. If valid it returns right away with the attempt number. If not, it records the problem, appends the bad reply as the assistant's turn, and appends a user message that says what was wrong.
problems.append(problem)
messages.append(("assistant", reply))
messages.append(("user", "That reply was rejected: " + problem + ". Reply again with only the corrected JSON."))03Step 3: three scenarios
A scripted fake model plays three roles. One is right the first time. One writes gold (not an allowed plan) and then fixes it after the feedback. One never produces JSON. The output shows attempts, the final plan, how many characters were sent in total, and whether the fallback was used.
04Step 4: see the cost grow
chars_sent for the repaired request is 273 against 67 for the first-time request, about four times as much, because the second attempt sends the prompt again plus the bad reply and the feedback (and the first attempt counts too). The second example puts token numbers and example prices on this.
05Step 5: choose the fallback deliberately
The third scenario returns free and says it was a fallback. In a real product ask who sees that value. If it is a plan used for billing, a silent default could cost money, so you might return an error or queue the request for a human instead.
Try it yourself
- 1
Predict chars_sent
The prompt is 67 characters long, so the first-time scenario sends 67. Predict whether the repaired scenario sends about double or more (it resends the prompt, plus the bad reply and the feedback). Run and compare.
- 2
A second repair
Change scenario 2 so its first two replies are bad (
gold, thenplatinum) and the third is good. How many attempts, and what happens tochars_sent? - 3
Move the limit
Set
max_attemptsto 2 and rerun. Which scenario changes, and what does the output show for it?
Code & diagrams
The fake model is scripted, so each scenario is repeatable. In production the model call is a real client behind the same function.
Expected output
fine first time attempts=1 plan=pro chars_sent=67 fallback=no
fixed after feedback attempts=2 plan=pro chars_sent=273 fallback=no
rejected: "plan" must be one of free, pro, got "gold"
never fixed attempts=3 plan=free chars_sent=537 fallback=yes
rejected: the reply is not valid JSON
rejected: the reply is not valid JSON
rejected: the reply is not valid JSONPrices are example numbers. Read your provider's price page and keep prices in configuration.
Expected output
1 attempt(s): 0.000240 dollars
2 attempt(s): 0.000545 dollars
3 attempt(s): 0.000915 dollars
1,000 requests with repairs: 0.2620 dollars
1,000 requests if all were fine: 0.2400 dollars
repairs add 9 percentBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Allow only one attempt
In the Python example change the default max_attempts=3 to max_attempts=1, run it, and compare the second scenario.
Myth vs fact
Myth
Retrying with the same request fixes bad output.
Fact
At temperature 0 the same input usually gives the same output. Without feedback a retry mostly wastes money.
Myth
More attempts always mean more successes.
Fact
After the second or third attempt the gain is tiny. A model that failed three times is rarely fixed by a fourth.
Myth
A fallback means the system is safe.
Fact
A fallback is only safe if the value is harmless and the failure is counted. A silent default can be a quiet wrong answer.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Track
first_try,repairedandfallbackas three counters. A rising repair rate after a model or prompt change is an early warning (Topic 11.4). - ▸
Put a cap on total time and money as well as attempts: a deadline for the whole operation, and the worst-case token budget checked before each call (Topics 6.2, 6.9).
- ▸
On long conversations you can send only the last bad reply and the feedback to the repair call instead of everything. Test that the quality holds, because it saves input tokens.
Remember this
- 1
Why this exists. Validation (Topic 4.5) finds the problem; it doesn't solve it. You have two options: give up on that request, or ask the model again. Models are good at fixing a specific complaint about their own output, so a retry with the error text succeeds far more often than a blind retry of the same prompt. Most reply failures are one-off slips, so one repair is usually enough.
- 2
The loop. Call the model, check the reply. If it is valid, return it. If not, append the bad reply and a short message with the problems to the conversation, and call again. Stop after a fixed
max_attempts. This looks like a network retry loop (Topic 6.8 teaches those later), but the retry reason is content rather than a network error, and the second request is different from the first. Recall from Topic 4.5: the problem list comes from your validator. - 3
Always have a maximum. A loop with no limit can run forever and spend without end. Two or three attempts is typical. If the third reply still fails, the model is unlikely to improve with a fourth. Pick the limit from your data: measure how many requests succeed on attempt 2 and 3, and stop where the gain flattens.
- 4
Then a fallback. After the last failed attempt the program needs a defined outcome: a safe default value (like plan
free), a marker such asneeds human review, or an honest error to the caller. Which one depends on the product. What matters is that it is chosen on purpose and that the result says it was a fallback, so counters and logs can show how often it happens (Phase 11). - 5
Retries cost money and time. Each attempt resends the whole conversation, including the rejected reply and your feedback, so attempt 3 is larger than attempt 1. The second example works the numbers with example prices: if 5% of requests need one repair and 1% two, the bill rises by about 9%. That is usually a good deal against failing, and a bad deal if the cause is a bad prompt. If the repair rate is high, fix the prompt or the schema instead of paying for retries.
- 6
Retry the right thing. Repairing content failures is different from retrying network errors (rate limits, timeouts), which Topic 6.8 handles with backoff and jitter. Do not mix the two counters: a 429 is not a model mistake. And do not retry a bad reply at temperature 0 with the same messages, because the same input usually gives the same output. Hosted models are not perfectly repeatable, but you cannot rely on a change. The feedback is what changes the result.
Explain it without notes
Why does sending the validation error back to the model help?
How do you choose max_attempts?
What should a fallback look like, and why must it be flagged?
What does a repair loop cost, and how would you reduce it?
Practice
Write retry_until_valid(model, check, max_attempts) where model() returns text and check(text) returns an error string or None. Return the first valid text, or None after max_attempts calls.
Write feedback(problems) that returns the message to send to the model: That reply was rejected: p1; p2. Reply again with only the corrected JSON.
Trade-offs
- ↔
More attempts raise the success rate and the worst-case cost and latency.
- ↔
Sending full history keeps context for the repair; sending only the last turn is cheaper but may lose it.
- ↔
A hard fallback keeps the product running but can hide model problems unless it is counted.
Done when you can
Done when you can write a loop with a maximum number of attempts.
Done when you can write feedback that names the problem, not just the failure.
Done when you can choose and flag a fallback on purpose.
Done when you can work out what repairs add to a bill.
Done when you can say why content repairs and network retries are separate.