Topic 9.6
Finding Where It Failed
In one line
A wrong answer can come from five different stages. A triage walks the stages in order and names the first one that broke, so each failure goes to the right fix instead of to guesswork.
Think of it like this
A parcel never arrives. The sender might not have posted it, the van might have missed your street, it might have been put at the bottom of the pile and not delivered that day, the driver might have run out of room and left it at the depot, or it might have arrived and sat unopened in the hall. You ask in order: was it posted, did it enter the network, did it get picked, did it fit on the van, did anyone open it? Each answer points at a different person to call.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Triage
- Sorting problems into causes so each goes to the right fix.
- Not indexed
- The right passage never entered the search index.
- Not retrieved
- It is in the index but was not among the results.
- Ranked too low
- It was returned, but below the k passages sent to the model.
- Cut by the budget
- It was in the top k but the prompt's token budget ran out before it.
- Ignored by the model
- It was in the prompt and the answer still lacks its fact.
- Bucket
- A group of failures with the same cause.
Step by step
01Step 1: record what each stage did
For every failing question you need: the expected passage, the full ranking returned (say 5 or 10), which passages made the prompt, and the answer. If the prompt builder drops passages, log which.
02Step 2: ask whether it was ever indexed
Look the passage up in the index or the ingestion log. If it is absent, nothing downstream can work, and no amount of re-ranking will help.
if expected not in INDEXED:
return "not indexed", expected + " is not in the index"03Step 3: ask whether search returned it
If it is indexed but missing from the returned list, the matching failed: the question and the passage do not look alike to the search. This is the vocabulary-mismatch failure from Topic 1.6 and the usual target of embeddings, hybrid search and query rewriting.
04Step 4: compare its rank with k
A rank above k means the model never saw it. A re-ranker or a larger k could rescue it. Note the rank in the verdict so you can see how far it was from the cutoff: rank 4 is a near miss, rank 40 is a different problem.
05Step 5: replay the prompt budget
Even inside the top k, the prompt may drop passages when tokens run out. Replay the packing: take passages in rank order, stop at the first that does not fit. If the expected one is not in the list sent, the budget cut it.
06Step 6: count and decide
Run the triage over every failing question and count verdicts. The example's counts are one each, which is what a small demo looks like. Real sets are lumpy: one stage usually owns most failures, and that is where to work first.
Try it yourself
- 1
Predict before you run
Change the budget to 70 tokens. Which case moves, and to which verdict? Work it out from the token counts, then run.
- 2
Make the ranking cutoff matter
Set
K = 5. Which verdicts change? What does that say about trading a bigger k against the prompt budget? - 3
Add a case
Add a case for D3 where the ranking is
["D3", ...]and the answer contains30 minutes. It should beok. Then change the answer so that it lacks the fact and see the verdict. - 4
Count your own
Write five questions your own handbook-style product might get wrong. For each decide the stage, then say what you would check to prove it.
Code & diagrams
K = 3 passages may be sent and the prompt may spend 40 tokens on sources (toy numbers). The token counts per passage are example values, not measured with a real tokenizer.
Expected output
case 1: ok: D8 was sent and used
case 2: not indexed: D12 is not in the index
case 3: not retrieved: D3 is not in the 5 results
case 4: ranked too low: D1 is rank 5, only the top 3 are sent
case 5: cut by budget: sent D10 (31 of 40 tokens)
case 6: ignored by the model: D9 was sent but the answer lacks '14 days'
stages: ok 1, not indexed 1, not retrieved 1, ranked too low 1, cut by budget 1, ignored by the model 1Stage First check Usual fix
not indexed Is the passage id in the index? Ingestion: parser, filters, missed files
not retrieved Is the id in the top 5 or 10? Hybrid search, query rewrite, embedding model, chunking
ranked too low Is its rank greater than k? Re-ranker, larger candidate list, larger k
cut by budget Did it fit when passages are packed? Smaller chunks, larger budget, better packing
ignored Is the fact in the answer? Prompt layout, citations, model choiceBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Check the answer first
In the Python tab, make the answer check the first test in triage: add if fact not in answer: return "ignored by the model", "answer lacks " + fact as the first lines of the function. Run.
Myth vs fact
Myth
A wrong answer means the model is weak.
Fact
In the example only one of five broken cases was the model's doing. The rest were decided before it saw a word.
Myth
If a passage is in the index, search will find it.
Fact
Indexed is only the first of five conditions. The login question's passage was indexed and still never came back.
Myth
More context fixes everything.
Fact
A larger k rescues 'ranked too low' and makes 'cut by budget' more likely. Each fix moves the pressure to the next stage.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the full ranking (ids and scores) for 10 or 20 results, not just the k sent. 'Ranked too low' needs the passages you did not send.
- ▸
Add an automated 'every source document has at least one chunk' check to ingestion (Phase 4). It makes 'not indexed' nearly disappear and is cheap.
- ▸
A passage can be both ranked too low and split across two chunks so neither chunk matches. Look at chunk boundaries when 'not retrieved' keeps happening for long documents (Topics 4.4 and 4.5).
- ▸
Triage verdicts make a good dashboard: count per stage per week. A sudden rise in 'not indexed' after a deploy is an ingestion regression, found in minutes instead of days (Topics 11.6 and 11.9).
Remember this
- 1
Why a triage. A bad answer is a symptom. 'Make the model better' is the usual reaction and it is often the wrong one. Walking through the stages in order, with ids and numbers you already have, turns the symptom into a cause and the cause into a specific fix.
- 2
The five stages. (1) Not indexed: the passage that answers the question never made it into the index (ingestion skipped it, a parser failed, a filter dropped it). (2) Not retrieved: it is in the index, but search did not return it at all. (3) Ranked too low: search returned it, but below the
kpassages that are sent. (4) Cut by the budget: it is within the topk, but the prompt budget ran out before it. (5) Ignored by the model: it was in the prompt, and the answer still lacks the fact. - 3
Check in order, name the first break. The order follows the data: a passage cannot be retrieved if it was never indexed, and cannot be used if it was never sent. The example checks 'not indexed' first, then 'not retrieved', then rank against
k, then the budget, and last whether the answer contains the expected fact. The first failing check is the stage to fix; later stages are not blamed for earlier damage. - 4
Reading the example. Six cases produce six different verdicts. D12 (the stolen-bike fee) was never ingested: not indexed. The login question's passage D3 is absent from the five results: not retrieved. The money-back question finds D1 at rank 5 while only 3 are sent: ranked too low. The scratched-bike question has D5 at rank 2, but the first passage already used 31 of 40 tokens and D5 (27) did not fit: cut by budget. The lost-jacket question had D9 in the prompt and an answer without '14 days': ignored by the model. The pass question is fine.
- 5
Each stage has its own fix. Not indexed: ingestion (Phase 4): find why the document was skipped, add a check that every source document has chunks. Not retrieved: better matching: hybrid search, query rewriting, a different embedding model, chunk size (Phases 2, 3, 6, 7). Ranked too low: a re-ranker, a larger first-stage candidate list, or a larger k (Phase 6, Topic 9.3). Cut by budget: smaller chunks, a larger budget or better packing (Topic 8.2). Ignored by the model: prompt layout, instructions to quote and cite, or a different model (Topics 8.1 and 8.3).
- 6
Count the buckets. One failure is an anecdote. Triage the whole set of failing questions and count: if most are 'ranked too low', a re-ranker is worth building; if most are 'not indexed', fix ingestion before anything else. The counts rank your to-do list by how many questions each fix could rescue.
- 7
What the triage needs from the product. It needs the information in the run record: the ids and scores of the results, which passages made the prompt, and the answer. If you did not log them you cannot triage after the fact (Topic 11.6). 'Not indexed' also needs a way to ask the index whether a passage exists, which is why Phase 10's delete audit and this check share a mechanism.
Explain it without notes
List the five stages in order and say what each means.
Why is the order of the checks important?
A question has the right passage at rank 5 with k = 3. Which stage is it, and name two fixes.
How would you decide where to spend a month of work on a failing RAG product?
Practice
Write stage_counts(verdicts) taking a list of verdict strings and returning a dict of counts. Print it for a list of six verdicts of your choice.
Write packed(ranking, tokens, k, budget) that returns the ids that fit, taking passages in rank order from the top k and stopping at the first that does not fit. Print the result for a ranking of three passages with token counts 31, 27 and 18 and a budget of 40.
Trade-offs
- ↔
Stop at the first break: a clear owner for every failure. Report every broken stage: more complete, and noisier.
- ↔
Logging long rankings: enough data to triage, more storage and privacy care.
- ↔
Automated triage: fast and consistent, only as good as the logged data and the rules you wrote.
Done when you can
Done when you can name the five stages and the usual fix of each.
Done when you can run a triage in pipeline order and name the first break.
Done when you can say what data a run record needs to support triage.
Done when you can turn verdict counts into a priority list.