Topic 6.7
Does It Pay Off?
In one line
Every extra stage costs code, time and money. Measure each stage's gain on your own questions with hit rate and reciprocal rank, and keep the simplest pipeline that is good enough.
Think of it like this
You add a turbo, a spoiler and wider tyres to a car, one at a time, and time a lap after each. The turbo takes three seconds off, the spoiler nothing, the tyres one second but they wear out in a week. Without the stopwatch you would have kept all three on faith.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Test set
- Real questions, each with the id of the passage that answers it. Used to measure, never to tune and report on the same data.
- Hit@1 / hit@3
- The number of questions whose right passage is first / in the top three.
- Reciprocal rank
- 1 divided by the place of the right passage (1, 0.5, 0.33, ...), or 0 if it is missing.
- Mean reciprocal rank (MRR)
- The average reciprocal rank over all questions.
- Baseline
- The simplest system you compare against, such as BM25 alone.
- Held-out
- Questions not used to design or tune the system, so the score is not flattered.
Step by step
01Step 1: list real questions with answers
Write down eight questions and the id of the passage that answers each. Include easy ones, paraphrases, exact-term questions and at least one that is hard for everything. In the example: ("Where do lost items go?", "D9").
02Step 2: put every system behind one function
Each system is a function from a question to a ranked list of passage positions: keyword_ranking, vector_ranking, hybrid_ranking, hybrid_rerank. The same loop then measures all of them (the interface idea from Topic 0.7).
03Step 3: find the right passage's place
find(ranking, gold) returns the 1-based place, or 0 when the passage is not in the list. Keyword lists can omit passages, so a missing answer must count as a miss.
def find(ranking, gold):
shelf = [HANDBOOK[i][0] for i in ranking]
return shelf.index(gold) + 1 if gold in shelf else 0
places = [find(system(question), gold) for question, gold in CASES]
hit1 = sum(1 for p in places if p == 1)
mrr = sum(1 / p for p in places if p) / len(places)04Step 4: read the table
Compare hit@1, hit@3 and MRR across systems. Then compare *pairs*: which questions did stage B fix compared with stage A, and which did it break? The last two lines of the example print exactly that.
05Step 5: add cost to the decision
Beside each row write latency and cost. The right system is the cheapest one whose numbers are good enough for your users, not the one with the highest row.
06Step 6: keep measuring
Re-run the table when the corpus, the models or the question mix change. Keep it in your test suite (Phase 9).
Try it yourself
- 1
Add a question of your own
Add
("Do I need a helmet at age 12?", "D4")and run. Do all four systems still put D4 first? What does the table say about the questions you add yourself? - 2
Find the cost of the first stage size
Change
first_sizeinhybrid_rerankto 3, then to 12. How does the last row change, and how many pair scores per question does each setting cost? - 3
Hand-check one MRR
For places
[1, 3, 0, 2], compute MRR by hand (add 1, 1/3, 0, 1/2 and divide by 4) before you run it in code.
Code & diagrams
Toy vectors and the TOY pair scorer, so only the method is real. The relation table of the scorer was written with these eight questions in mind: treat 8/8 as a demonstration, not a result.
Expected output
keyword hit@1 4/8 hit@3 7/8 MRR 0.671
vector hit@1 2/8 hit@3 5/8 MRR 0.476
hybrid hit@1 6/8 hit@3 6/8 MRR 0.796
hybrid + re-rank hit@1 8/8 hit@3 8/8 MRR 1.000
re-ranking fixed: D4, D12
re-ranking broke: nothingBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Hand the re-ranker too few candidates
In the Python tab, in hybrid_rerank, change first_size=8 to first_size=3 and run.
Myth vs fact
Myth
A fancier pipeline is always better.
Fact
Each stage helps some questions and can hurt others. In the example hybrid has a lower hit@3 than plain keyword search. Measure.
Myth
Eight questions are enough to decide.
Fact
Eight shows the method. Decisions need dozens to hundreds of real questions, because one question moving is a large percentage on a small set.
Myth
If the demo looks right, the system is right.
Fact
Demos use the questions you thought of. A test set built from real user questions finds failures you would never try.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Report paired differences: for each question, was system B better, equal or worse than A? With few questions, "fixed 2, broke 0" is more informative than two averages. With more data, use a paired test or bootstrap intervals.
- ▸
Split the questions by type (exact term, paraphrase, multi-fact, unanswerable). A stage that helps one type and hurts another is a candidate for routing (Topic 7.6), not for an all-or-nothing decision.
- ▸
Do not tune on the questions you report on. Keep a held-out set, or at least look at the final numbers on questions you did not use to choose
first_sizeorlam. - ▸
Re-ranker gains are often largest when hit@3 is already high: the right passage is in the shortlist and just not first. If hit@3 is low, fix the first stage (chunking in Phase 4, queries in Phase 7) before buying a bigger re-ranker.
Remember this
- 1
Why this exists. Hybrid search, re-ranking and diversity each add to what you build, run and pay for. Each *sounds* helpful. Only a measurement on your own questions says whether it is. The good news: measuring costs one afternoon with a small test file of real questions, each with the id of the passage that answers it.
- 2
The three numbers. Hit rate at 1 (
hit@1): in how many questions the right passage is first. Hit rate at 3 (hit@3): in how many it is in the top three. Mean reciprocal rank (MRR): for each question take 1 divided by the right passage's place (1, 0.5, 0.33, ...; 0 if absent) and average. MRR rewards moving the right passage up, even from 6th to 4th. Phase 9 builds these properly; this topic uses them by hand. - 3
The example, read carefully. Eight questions, four systems. Keyword: hit@1 4/8, hit@3 7/8, MRR 0.671. Vector (the toy): 2/8, 5/8, 0.476. Hybrid: 6/8, 6/8, 0.796. Hybrid with the toy re-ranker: 8/8, 8/8, 1.000. Re-ranking fixed the helmet question (D4) and the theft question (D12) and broke nothing.
- 4
What the numbers also say. Hybrid is better than either single ranker at hit@1 (6/8 against 4/8 and 2/8) but its hit@3 (6/8) is *lower* than keyword search's (7/8): fusing with a weak vector side pulled a passage out of the top 3 that keyword search had there. A stage that helps on average can hurt a few questions, so look at which questions moved, not only the totals.
- 5
Do not believe this table. There are only 8 questions, the vector side is a toy, and the re-ranker's word-relation table was written by looking at these very questions. A perfect 8/8 is therefore a demonstration of *how to measure*, not evidence about real re-rankers. With a small test set, a difference of one or two questions can be noise: on 8 questions each question is worth 12.5 percentage points of hit rate, on 200 questions only 0.5. Collect more real questions (from logs, from users) before deciding.
- 6
When plain is right. Use keyword only when the corpus is small or full of exact identifiers, the questions repeat the document's wording, or you need explainability and zero extra infrastructure. Use vectors only when questions are paraphrases of the content and there are almost no codes or names. Add hybrid when your test set contains both kinds of question. Add a re-ranker when hit@3 is high but hit@1 is low (the right passage is near the top but not first) and your latency budget has room. The decision is per stage and per corpus.
- 7
Cost goes in the table too. For each system write down the latency (p50 and p95), the money per 1,000 questions and what you must keep running. A stage that adds 2 points of hit@1 for 300 ms and a second service may not be worth it for a help-centre chatbot but is for a legal search tool.
Explain it without notes
Define hit@1, hit@3 and MRR, and say what MRR shows that hit@1 does not.
In the example, hybrid had a lower hit@3 than keyword search. What does that teach?
When would you choose plain BM25 over hybrid search?
Why are the example's 8/8 results not evidence that re-ranking will work for you?
Practice
Write mrr(places) where places is a list of 1-based places (0 meaning missing) and print it for [1, 3, 0, 2] with 3 decimals.
Write compare(before, after) that takes two lists of places and returns the number of questions fixed (not first before, first after) and broken (first before, not first after). Print both for before = [1, 3, 2, 1] and after = [1, 1, 2, 2].
Trade-offs
- ↔
Simplest system: cheapest to run and explain, may leave gain on the table.
- ↔
Every added stage: possible gain on some questions, plus cost, latency and a new failure mode.
- ↔
Small test set: quick to build, noisy. Large test set: trustworthy, costs time to collect and label.
Done when you can
Done when you can compute hit@1, hit@3 and MRR by hand for a list of places.
Done when you can compare two systems question by question.
Done when you can say when BM25 alone or vectors alone is the right answer.
Done when you can say why a toy result proves nothing about your own data.