PHASE 6Intermediate ~27 min· topic 5 of 9

Topic 6.5

The Retrieve-Then-Rerank Pipeline

In one line

Run a fast, wide first stage to collect candidates, then a slow, sharp second stage on only those. Two sizes matter: how many candidates you hand over, and how many you keep. Measure both.

Think of it like this

Casting a film. Open auditions let hundreds of people read one line each (quick and cheap, you see everyone). Callbacks give 12 people a long scene with the other actors (slow and revealing, so you can only afford a few). Too few people at the callback and your best actor never gets a second look. Too many and callbacks take all week.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Retrieve-then-re-rank: A two-stage design: a fast first stage returns candidates, a slower second stage re-orders them.
First stage: The fast, wide search (here: hybrid keyword and vector search).
Second stage: The slower, sharper re-ranker that scores each candidate with the question.
Recall at N: The share of questions whose right passage is somewhere in the first N results.
Precision: How many of the results you keep are actually relevant. Re-ranking aims to raise it at the top.
first_size / final_k: How many candidates stage one passes on, and how many passages the pipeline finally keeps.
Fallback: A safe behaviour when a stage fails, such as keeping the first-stage order.

Step by step

01Step 1: stage one, wide and cheap

hybrid_ranking(question)[:first_size] gives the candidates: keyword and vector rankings fused with RRF, cut at first_size. The example's first line prints the 8 candidates for the helmet question: D5 D1 D3 D6 D4 D9 D8 D10. The right passage, D4, is fifth.

02Step 2: stage two, narrow and sharp

rerank(question, candidates) scores each candidate with the pair scorer and sorts by that score, keeping first-stage order for ties. D4 gets 0.87 and the rest 0.00 in this example, so D4 moves from fifth to first.

pipeline.pywhole filepython

def retrieve_then_rerank(question, first_size, final_k):
    candidates = hybrid_ranking(question)[:first_size]  # stage 1: wide and cheap
    return rerank(question, candidates)[:final_k]       # stage 2: narrow and sharp

03Step 3: keep final_k

The pipeline returns the first final_k of the re-ordered list: here 3 passages for the prompt (D4 D5 D1 for the helmet question).

Step 3: keep final_kdiagram

Rendering diagram…

04Step 4: sweep first_size

Run the pipeline for sizes 3, 5, 6, 8 and 12 over three questions and count how many have the right passage first. Pair calls and time grow linearly with the size, so the table shows both the gain and its price.

05Step 5: pick the size and a margin

Take the smallest size that works on your test questions and add some room, because your sample is small. Re-measure when the corpus or the first stage changes.

06Step 6: plan for failure

A real re-ranker is another service. Give it a timeout and fall back to the first-stage order if it fails (recipe in Topic 6.9). A slightly worse answer beats an error.

Try it yourself

1
Shrink the second stage
Set final_k to 1 and 5 for the helmet question. What changes in the prompt you would build, and which choice is safer if the re-ranker is slightly wrong?
2
Add a question
Add ("Where do lost items go?", "D9") to HARD and rerun the sweep. At which first_size does it become right? Is the answer still 6?
3
Change the price
Set PAIR_MS to 25 and compute the time for sizes 6 and 30. Is a size-30 shortlist affordable at a 300 ms budget?

Code & diagrams

Retrieve, then re-rank: choosing the first stage's size New tab

TOY pair scorer and toy vectors, as in Topic 6.4. The first-stage ranks of the three questions (1st, 5th, 6th) come from running this hybrid. A real pipeline's numbers will differ: measure yours.

Expected output

Q: What is the age limit for helmets?
stage 1, hybrid, 8 candidates: D5 D1 D3 D6 D4 D9 D8 D10
stage 2, pair scores:          D4=0.87  D5=0.00  D1=0.00  D3=0.00
final 3:                       D4 D5 D1
first stage  pair calls  gold first  time per question
          3           3  1 of 3     24 ms
          5           5  2 of 3     40 ms
          6           6  3 of 3     48 ms
          8           8  3 of 3     64 ms
         12          12  3 of 3     96 ms

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Hand over too few candidates

In the Python tab, in retrieve_then_rerank, replace [:first_size] with [:3] and run.

terminal

$ Run

── what you'll see ──

Q: What is the age limit for helmets?

stage 1, hybrid, 8 candidates: D5 D1 D3 D6 D4 D9 D8 D10

stage 2, pair scores: D4=0.87 D5=0.00 D1=0.00 D3=0.00

final 3: D5 D1 D3

first stage pair calls gold first time per question

3 3 1 of 3 24 ms

5 5 1 of 3 40 ms

6 6 1 of 3 48 ms

8 8 1 of 3 64 ms

12 12 1 of 3 96 ms

Myth vs fact

Myth

A better re-ranker makes the first stage unimportant.

Fact

The re-ranker can only re-order what it receives. A weak first stage that leaves out the right passage caps the whole pipeline.

Myth

Bigger shortlist is always better.

Fact

Past the point where the right passage is in the shortlist, a bigger one only adds latency and cost, and gives a re-ranker more chances to promote a wrong passage.

Myth

Re-ranking is optional polish.

Fact

On some corpora it is the biggest single gain, on others it changes almost nothing. Measure on your questions (Topic 6.7).

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Candidates should be deduplicated before re-ranking, or you pay twice to score copies (Topic 6.6). Do cheap filtering (access control, metadata, duplicates) before the expensive stage.
▸
Re-ranker scores are often more usable as an evidence signal than fused scores: a low best score for every candidate suggests the question is not answerable from the corpus. Calibrate a threshold on labelled questions per model (Topic 8.4), and never carry it across models.
▸
Log both stages: first-stage rank and re-rank score for each passage that reached the prompt. When the answer is wrong you can tell whether stage one missed the passage or stage two demoted it.
▸
Budget by percentile. Mean latency hides tail latency: hosted re-rankers have slow requests, and the user waits for the slowest one. Set a timeout near your p95 and fall back past it.

Remember this

1
Why this exists. Topic 6.4 showed that a cross-encoder is accurate but costs one model run per pair. Topic 6.3 showed that hybrid search is fast. A retrieve-then-re-rank pipeline uses each for what it is good at: the first stage is wide and cheap and aims for recall (the right passage is somewhere in the list); the second stage is narrow and sharp and aims for precision (the right passage is at the top).
2
Two numbers to choose. first_size is how many candidates stage one hands over. final_k is how many passages stage two keeps for the prompt. final_k comes from your prompt budget and answer needs (Phase 8). first_size is a trade-off you measure: too small and the right passage never reaches the re-ranker, too big and you pay for pair scores that change nothing.
3
Reading the sweep. The example runs the pipeline on three questions where the right answer is at various depths in the hybrid list: the monthly pass is first, the helmet passage fifth, the theft passage sixth. With first_size 3 only one of three questions has the right passage first (the monthly pass). At 5 the helmet question is fixed (2 of 3). At 6 all three are right, and 8 or 12 add nothing but time: 64 ms and 96 ms of pair scoring per question at the assumed 8 ms. The best size is the smallest one that is right most of the time on your questions, plus a safety margin.
4
The latency budget. Total time is first-stage search, plus first_size pair scores (divided by how many your hardware scores in parallel), plus network time if the re-ranker is a hosted API. Cost follows the same shape: hosted re-rankers usually bill per request or per set of documents (check your provider), so a bigger first_size costs more money on every question.
5
Real re-rankers. Two were checked for this course. BAAI/bge-reranker-v2-m3 is an open multilingual model under the Apache 2.0 licence that you run yourself (Topic 6.4 shows its Python call). Cohere Rerank is a hosted API: POST https://api.cohere.com/v2/rerank with a model, a query, a list of documents and an optional top_n; it returns results with each document's index and a relevance_score between 0 and 1. Its docs recommend no more than 1,000 documents per request. Topic 6.9 shows the call wired in with a fallback. Prices and model lists change, so check the provider's page.
6
Where things go wrong. If the first stage is poor, the re-ranker polishes the wrong shortlist. If the re-ranker is slow or down, every question suffers unless you fall back to the first-stage order. And re-ranking is not always an improvement, which is why the next topics measure it.

Explain it without notes

Explain the two stages of retrieve-then-re-rank and what each is optimised for.

How do you choose the first stage's size?

What can go wrong if the re-ranker is a remote service?

A re-ranker did not improve results. List two possible reasons.

Practice

Write recall_at(rankings, golds, n) that returns how many questions have the gold id within the first n of its ranking. Print it for n = 1 and n = 3 given rankings = [["a", "b", "c"], ["b", "a", "c"], ["c", "b", "a"]] and golds = ["a", "a", "a"].

Write budget_ms(search_ms, pair_ms, candidates, parallel) returning the stage-one time plus the re-ranking time when parallel pairs are scored at once (round the number of rounds up). Print it for 5, 8, 30 with parallel 1, and with parallel 10.

Trade-offs

↔
Small first_size: cheap and fast, risks leaving the right passage out.
↔
Large first_size: higher recall, more latency and cost, more chances to promote a wrong passage.
↔
Hosted re-ranker: no hardware to run, but per-request cost, network latency and a dependency.

Done when you can

Done when you can draw the two-stage pipeline and name what each stage optimises.
Done when you can pick first_size from a recall measurement.
Done when you can compute the latency budget for a shortlist.
Done when you can describe the fallback for a failing re-ranker.

Command Palette

Words you'll meet

Step by step

01Step 1: stage one, wide and cheap

02Step 2: stage two, narrow and sharp

03Step 3: keep final_k

04Step 4: sweep first_size

05Step 5: pick the size and a margin

06Step 6: plan for failure

Try it yourself

Shrink the second stage

Add a question

Change the price

Code & diagrams

Break it on purpose

Hand over too few candidates

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can