Topic 6.4
Re-ranking with Cross-Encoders
In one line
An embedding squeezes each passage into one vector before it ever sees the question. A cross-encoder reads the question and one passage together and scores the pair: much sharper, far too slow for the whole index, so it re-orders only a shortlist.
Think of it like this
A hiring manager first skims one-line summaries of 1,000 CVs, written long before this job existed, and picks 20. Then she interviews those 20 with the job description in front of her, asking questions about this specific role. The skim is fast but blunt; the interview is sharp but takes an hour per person, so she would never interview all 1,000.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Bi-encoder
- A model that turns the question and each passage into vectors separately. Passage vectors can be computed in advance.
- Cross-encoder
- A model that reads the question and one passage together and outputs one relevance score for the pair.
- Re-ranker
- A second-stage model that re-orders a shortlist of candidates from the first stage. A cross-encoder is the usual kind.
- Pair
- One question together with one passage. A cross-encoder scores pairs.
- Shortlist
- The few candidates the first stage hands to the re-ranker.
- Toy scorer
- The hand-written stand-in for a cross-encoder used in this course's examples. Not a real model.
Step by step
01Step 1: what the bi-encoder cannot see
Each passage was turned into a vector before the question existed. Two passages that differ in one decisive word can end up almost on top of each other, and the question's exact phrasing never gets to interact with the passage's phrasing.
02Step 2: score the pair together
The toy scorer takes the question and one passage and returns one number from 0 to 1: the average, over the question's content words, of 1.0 for a word present in the passage, 0.8 for a related word and 0 otherwise. The code below is an excerpt: the full function, with tokenize, stem, STOP and the related-word table, is in the first example.
def pair_score(question, passage):
"""TOY cross-encoder: reads the question and ONE passage together and returns one number from 0 to 1."""
wanted = [stem(w) for w in tokenize(question) if w not in STOP]
words = {stem(w) for w in tokenize(passage)}
...03Step 3: compare with one vector per text
For the helmet question, the toy vector side puts D4 at place 4 (the toy knows none of the question's words, so ties keep handbook order). The pair scorer puts D4 first, with 0.87. For the theft question the gap is larger: place 9 against place 1.
04Step 4: count the cost
A pair needs a model run each time, so the time grows with the number of passages scored. The second example prints the arithmetic for 10 up to 100,000 passages at an assumed 8 ms per pair.
05Step 5: use it on a shortlist only
Score 30 candidates, not 100,000. The first stage must still bring the right passage into those 30 (Topic 6.5 measures how many are needed).
06Step 6: see what a real one looks like
An open re-ranker such as BAAI/bge-reranker-v2-m3 (Apache 2.0 licence, multilingual) takes a [query, passage] pair and returns a score; the model's page describes it as taking the question and document as input and directly outputting similarity, not embeddings.
from FlagEmbedding import FlagReranker
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)
score = reranker.compute_score(["What is the age limit for helmets?", "Riders under 16 must wear one."])Try it yourself
- 1
Add a relation
Add
("discount", "credit")toRELATEDin the first example and score the question"Is there a discount if I cancel late?"against all 12 passages. Does the right passage reach the top? What did you just do, in terms of a real re-ranker's training? - 2
Change the price
In the cost example set
PAIR_MSto 40 (a bigger model) and the shortlist to 100. How long does one question take now? - 3
Order of a tie
Find two passages with the same pair score for one question. Which is returned first? Why is a defined tie rule important for tests?
Code & diagrams
TOY: the pair scorer is a hand-written stand-in for a cross-encoder (its word-relation table is written out in the code). A real one learns such relations from data.
Expected output
Q: What is the age limit for helmets?
one vector per text, top 3: D1 D2 D3
pair scorer, top 3: D4=0.87 D11=0.27 D1=0.00
D4 is at: vector 4, pair scorer 1
Q: What happens if someone steals the bike?
one vector per text, top 3: D7 D11 D10
pair scorer, top 3: D12=0.90 D5=0.50 D6=0.50
D12 is at: vector 9, pair scorer 1The 8 ms and 5 ms are example numbers, not measurements. Measure your own model and hardware.
Expected output
passages scored time at 8 ms per pair
10 80 ms (0.1 s)
50 400 ms (0.4 s)
100 800 ms (0.8 s)
1000 8000 ms (8.0 s)
100000 800000 ms (800.0 s)
vector search, then 30 pairs: 245 ms
re-ranking the whole index of 100000 would take 3265 times as longBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Remove the learned relations
In the Python tab of the first example, replace the line elif any((w, p) in RELATED or (p, w) in RELATED for p in words): with elif False: and run.
Myth vs fact
Myth
A cross-encoder can replace the vector index.
Fact
It needs one model run per passage per question. Over a whole index that is far too slow, so it re-orders a shortlist that a fast first stage provides.
Myth
A re-ranker can find passages the first stage missed.
Fact
It only sees the candidates it is given. If the right passage is not in the shortlist, no re-ranker can bring it back.
Myth
Bigger re-ranker means better results on my data.
Fact
Larger models are slower and costlier, and only your own test set shows whether the gain is worth it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Pair scoring is batchable: a GPU scores many pairs at once, so wall-clock time grows slower than N. Hosted APIs take a list of documents in one request. Sequential per-pair calls over the network are the slow way.
- ▸
Input length matters. A cross-encoder reads question and passage together up to a token limit, and text beyond it is cut. Long chunks need splitting or the limit raised; one hosted API documents a default per-document truncation, so check your provider.
- ▸
Scores from different re-rankers are not comparable, and some are raw numbers rather than 0 to 1 (the BGE model's
compute_scorecan apply a sigmoid withnormalize=True). Never reuse a threshold from one model on another. - ▸
Latency budget: first-stage search in milliseconds, plus candidates x pair time divided by batch parallelism, plus network time for hosted models. Put the re-rank time in your tracing next to the search time (Phase 11).
Remember this
- 1
Why this exists. An embedding model (a bi-encoder) turns every passage into one vector in advance, without knowing what the question will be. That vector must serve every possible question, so details get squashed: whether the age limit is *under* 16 or *over* 16, the word "not", the exact number, which of two nearly identical passages answers the question. The question and the passage are only compared at the very end, as two points in space.
- 2
What a cross-encoder does. It is a model that takes the question and one passage together as a single input and produces one relevance score. Because both texts are inside the model at once, every word of the question can be compared with every word of the passage. It can notice that "age limit" in the question matches "under 16" in the passage, or that "steals" matches "stolen".
- 3
Far more accurate, far slower. The price: nothing can be computed in advance, because the model needs the question. Scoring N passages means N model runs at question time. In the cost example, with an assumed 8 ms per pair, scoring 1,000 passages takes 8.0 seconds and 100,000 takes 800 seconds. A bi-encoder search over the same index takes milliseconds because the passage vectors already exist.
- 4
So it only re-orders a shortlist. The standard design is two stages: a fast first stage (hybrid search, Topic 6.3) picks the top 20 to 100 candidates, and the cross-encoder re-scores only those. With a shortlist of 30, one question costs 5 ms of search plus 240 ms of pair scoring, 245 ms in the example. Scoring the whole 100,000-passage index would take 3,265 times as long. A cross-encoder can fix the *order* of candidates; it cannot rescue a passage the first stage left out.
- 5
A toy stand-in. A real cross-encoder cannot run inside these examples, so the examples use a toy pair scorer, clearly not a real model. It reads the question and a passage together, ignores small words, strips a plural
s, and uses a hand-written table of related word pairs ("age" with "under", "steal" with "stolen"). A real cross-encoder *learns* such relations from huge numbers of labelled question-passage pairs. For "What is the age limit for helmets?" the toy ranks D4 first with 0.87 while the toy vector puts D4 fourth; for the theft question it ranks D12 first and the vector puts it ninth. - 6
Be honest about the toy. The relation table was written by looking at these questions, so its good results are *not evidence* that a real re-ranker will do as well on yours. Only your own test questions (Topic 6.7) can tell you that. Real re-rankers exist as open models you run yourself and as hosted APIs; Topic 6.5 names two that were checked.
Explain it without notes
How does a cross-encoder differ from a bi-encoder, and why is it more accurate?
Why is a cross-encoder too slow to run over the whole index?
What happens to a relevant passage that the first stage did not return?
Why shouldn't you trust the toy scorer's results as evidence about real re-rankers?
Practice
Write rerank_ms(n, pair_ms) that returns the milliseconds to score n pairs sequentially. Print it for n = 30 and n = 1000 with pair_ms = 8.
Write order_by_pair(question, candidates, scorer) that returns the candidates sorted by scorer(question, candidate), best first, keeping the original order for ties. Print the result for candidates ["a", "bb", "cc"] with a scorer that returns the length of the candidate.
Trade-offs
- ↔
Cross-encoder: sharp ordering of a shortlist, at one model run per candidate per question.
- ↔
Larger shortlist: more chance the right passage is in it, linearly more re-ranking time and cost.
- ↔
Open model you host: control and no per-call fee, but you run the hardware. Hosted API: easy, but network latency, per-request cost and data leaving your system.
Done when you can
Done when you can explain the difference between a bi-encoder and a cross-encoder in two sentences.
Done when you can work out the re-ranking time for a shortlist size and a per-pair time.
Done when you can say why a re-ranker cannot fix recall.
Done when you can say why the toy scorer proves nothing about real models.