Topic 13.4
"The Answers Are Wrong": Debugging Interviews
In one line
A method for the most common RAG interview question: reproduce the failure, find the stage that lost the evidence, fix that stage, and add a test. Four realistic cases are triaged by code.
Think of it like this
A plumber called to a house where the tap is dry. A bad plumber replaces the tap. A good one checks the stop valve, then the pipe, then the tap, and stops at the first place where the water is missing. A RAG system is a pipeline too, and 'the answers are wrong' is the dry tap.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Trace
- A record of what the system did for one question: ids, scores, prompt and answer.
- Stage
- One step of the pipeline: index, retrieve, rank, pack, generate.
- Regression test
- A saved question and expected result that fails if an old bug returns.
- Reproduce
- Make the failure happen again on demand, with the same inputs.
- Ground truth passage
- The passage you know contains the answer, used to check retrieval.
- Ranked too low
- Present in the results but below the number of passages you keep.
Step by step
01Step 1: reproduce and keep the trace
Run the failing question again and capture the retrieved ids and scores, the prompt and the answer. If you cannot reproduce it, make the logging better first (Topic 11.6).
02Step 2: name the ground truth passage
Decide which passage should answer. Without it you cannot tell a retrieval failure from a generation one. This is the first golden-set row.
03Step 3: walk the stages in order
Is the passage indexed? In the ranking? Within k? In the prompt? In the answer? Stop at the first no. The triage function in the example is exactly this list.
04Step 4: fix one thing
Change the single stage you found. If the fix needs two changes, make them one at a time and re-run the trace between them.
05Step 5: run the golden set
A fix that helps one question can hurt others. Re-run the whole set and compare hit rate and refusal rate before and after (Topic 9.7).
06Step 6: add the case
Add the question, the expected passage and the stage you found. The suite now remembers the failure.
Try it yourself
- 1
Add a fifth case
Add a case where the passage is retrieved at place 1 but the budget sent only the other sources. Predict the stage, then run. What does that tell you about the ordering of the sources in the prompt?
- 2
Swap two checks
Move the 'not retrieved' check above the 'not indexed' check. Which case now gets a wrong stage? Why is the order part of the correctness?
- 3
Use the lab
Switch the ranker in the lab and note which one puts the password passage first. Which stage's fix does that correspond to: query, ranker or index?
Code & diagrams
Real systems fill these fields from the trace of each failing question. The order of the checks is the point: a passage must be indexed before it can be retrieved.
Expected output
brand-new policy page: not indexed
fix: check the loader, its filters and the last ingestion run
login question: ranked too low
fix: add a re-ranker or raise k
damage report: cut by the budget
fix: shorten the chunks or raise the context budget
lost items: ignored by the model
fix: fix the prompt, then check the order of the sourcesTry: Press Search with the keyword ranker. The password passage should be first for the login question, and it is third: its BM25 score is about 1.45, behind two passages that share small words such as 'in' and 'get'. Switch to the vector ranker, where it is first with a cosine of about 0.47 against 0.16 for the next passage. Then try hybrid: where does it come out, and why? Say which stage of the pipeline you just repaired.
Goals
- The password reset passage is first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Skip the indexed check
In the Python tab, delete the two lines if expected not in indexed: and return "not indexed", then run.
Myth vs fact
Myth
A wrong answer means the model needs improving.
Fact
In the triage the model is blamed correctly only for the last case. The other three failed before the model read anything.
Myth
If the right passage is retrieved, the system is right.
Fact
It may still be below k, cut by the budget or ignored. Each is a separate stage with a separate fix.
Myth
Tuning on the failing question is enough.
Fact
Without the whole golden set you cannot see what you broke elsewhere.
Interview problem
The problem
"The answers got worse after last week's release"
Your RAG assistant for employee benefits was fine last week. Users now report more wrong answers. You changed the chunker (smaller chunks) and upgraded the embedding model in the same release. What do you do?
You're given
- You have the old and new indexes, a golden set of 120 questions and request logs.
- Users complain about a few topics, not all.
- You must decide within a day whether to roll back.
The interviewer follows up
Hit rate is the same overall, but one topic dropped a lot. Is the release safe?
The correct passage is retrieved but the answer is still wrong. Now what?
You cannot reproduce the bug.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the five facts per question (expected unknown in production, but retrieved ids with scores, prompt token count, ids actually sent, answer, citations). With them, triage of a bad answer takes minutes instead of days.
- ▸
In production you rarely know the expected passage. Sample thumbs-down questions, ask a human to name the passage and add them to the golden set (Topic 9.8). That is the cheapest source of hard cases.
- ▸
Intermittent failures often come from changing data: the passage was indexed yesterday and removed by a sync today. Compare chunk timestamps with the failure time before touching the model.
Remember this
- 1
Why a method matters. The visible symptom, a wrong answer, can come from five different stages, and the fix for one is useless for another. A bigger model does not help if the passage was never indexed. A new chunker does not help if the model ignores a sentence that is in the prompt. The method finds the stage first.
- 2
The method: reproduce, locate, fix, test. (1) Reproduce with the exact question, the same data and the same settings, and keep the trace: retrieved ids, scores, the prompt and the answer. (2) Locate the stage by walking the pipeline in order and stopping at the first one that lost the evidence. (3) Fix that stage only. (4) Add the question to the golden set so the regression cannot come back (Topic 9.7).
- 3
The five stages, in order (Topic 9.6). Not indexed: the passage is not in the store at all (a loader skipped it, a filter removed it, ingestion failed). Not retrieved: it is indexed but not in the results. Ranked too low: it is in the ranking but below the cut at k. Cut by the budget: it was inside k but did not fit in the prompt. Ignored by the model: it was in the prompt and the answer still lacks the fact.
- 4
What each stage points to. Not indexed: the loader, filters, last ingestion run (Phase 4). Not retrieved: the query, the tokenizer, the embedding model, a missing ranker (Phases 1, 2, 6, 7). Ranked too low: a re-ranker, a larger k, fusion weights (Phase 6). Cut by the budget: chunk size, the context budget and ordering (Topic 8.2). Ignored: the prompt layout, source order, or a conflicting source (Topics 8.1 and 8.5).
- 5
Do not start with the model. The cheapest test is to paste the correct passage into the prompt by hand. If the answer is right, generation is fine and retrieval is the problem. If it is still wrong, look at the prompt and the instruction. This one experiment splits the pipeline in half.
- 6
Common traps. Looking only at the final answer, never the retrieved ids. Fixing several things at once, so nothing is learned. Tuning on the one failing question and breaking ten others, which only the golden set shows. Blaming the embedding model for a failure that is really a missing document.
Explain it without notes
Describe the four-step method for 'the answers are wrong'.
Name the five stages where evidence can be lost and what each points to.
What single experiment splits retrieval problems from generation problems?
Why add the question to the golden set after fixing it?
Practice
Write place(ranking, expected) that returns the 1-based place of expected in ranking, or 0 when it is missing. Print the place of 'D3' in ['D1', 'D10', 'D3'] and in ['D1'].
Given the triage results ['ranked too low', 'ranked too low', 'cut by the budget', 'not indexed'], print each stage with its count, most frequent first and ties in alphabetical order.
Trade-offs
- ↔
Logging every stage: makes triage quick, at the price of storage and a need for redaction (Topic 10.5).
- ↔
Triage by code versus by eye: code is repeatable on many cases, eyes catch what the checks do not model.
- ↔
Fixing the symptom quickly versus the stage properly: a quick prompt patch may hide a retrieval problem that returns next month.
Done when you can
Done when you can state the method (reproduce, locate, fix, test) from memory.
Done when you can list the five stages in order with a fix for each.
Done when you can explain the paste-the-passage experiment.
Done when you can say what to log so triage is possible later.