Topic 6.1
Why One Ranker Is Not Enough
In one line
Keyword search finds exact words, codes and numbers. Vector search finds the same meaning in different words. Each one confidently misses what the other finds, so serious search runs both.
Think of it like this
You ask two librarians for help. The first one knows every title and word on every spine, so if you say "ERR-4012" or "the 150 dollar fee" she walks straight to the shelf, but if you say "something about getting back into my account" she looks puzzled. The second one understands what you mean and finds the right book from a vague description, but he is hopeless with exact codes and numbers. Asking both, and trusting a book that both of them pick, is better than relying on either one.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- BM25 and cosine
- BM25 is the keyword score (Topic 1.4). Cosine similarity is the vector score: how close two embeddings point (Topic 2.2). Both are just numbers a ranker sorts by.
- Keyword search (lexical)
- Ranking by the words a passage shares with the question. BM25 is the standard formula (Topic 1.4).
- Vector search (semantic)
- Ranking by how close a passage's embedding is to the question's embedding (Phase 3).
- Hybrid search
- Running keyword and vector search together and merging their rankings into one list.
- Exact term
- A word that must match literally to be useful: an error code, a product name, an amount.
- Paraphrase
- The same meaning in different words, such as "login" and "password".
- Rank
- A passage's place in a list: 1 is the best, 2 is next, and so on.
- Toy embedder
- A tiny hand-made stand-in for an embedding model, used so examples can run anywhere. Not a real model.
Step by step
01Step 1: one question, two lists
Run the same question through the two rankers you already have. Each returns every passage in an order. Keyword search leaves out passages that share no word with the question. Vector search ranks every passage, even bad ones, because every vector has some similarity to every other.
keyword = keyword_ranking(question) # only passages that share a word
vector = vector_ranking(question) # every passage, best first02Step 2: a question keywords win
"150 dollar fee" contains a number that appears in exactly one passage. Keyword search puts D12 first. The toy vector side knows nothing about 150 and puts the monthly pass first, and D12 is third. The real model in the lab behaves the same way.
03Step 3: a question vectors win
"I forgot my login, how do I get back in?" shares no meaningful word with the password passage. The toy embedder files login, forgot and password under the same "account" dimension, so D3 comes first. Keyword search puts D3 in third place.
04Step 4: merge the lists
rrf([vector, keyword]) gives each passage a score from its place in each list (Topic 6.3 explains the formula). For the money question, D1 is second in both lists and ends up first in the merged list.
05Step 5: read where the right answer landed
The last line of each block shows the place of the right passage in the keyword list, the vector list and the hybrid list. Across the four questions, hybrid is first every time, while each single ranker is first on only some of them.
06Step 6: try it with the real model
The search lab below runs the real embedding model in your browser. It starts on the "150 dollar fee" question with the vector ranker, where the right passage is third. Switch the ranker to Keyword and then to Hybrid and watch it move to first.
Try it yourself
- 1
Predict, then run
Before running the example, add the question
"How long does an e-bike battery last?"with goldD7toCASES. Which of the three lists do you think puts D7 first? Run it and check. - 2
Find a question only one ranker answers
Write three questions of your own about the handbook. For each, say before running whether keywords or vectors should win. Which of your guesses were wrong?
- 3
Lab: switch the ranker
In the search lab, solve the challenge by changing only the ranker. Then keep the ranker on Hybrid and try
"monthly pass": does hybrid ever do worse than the better of the two single rankers on these questions?
Code & diagrams
The vector side is a toy embedder (five hand-made meaning dimensions), so its numbers are not a real model's. The lab below uses the real one.
Expected output
Q: What does the monthly pass cost?
keyword D8 D10 D3 | vector D8 D1 D12 | hybrid D8 D12 D10
D8 is at: keyword 1, vector 1, hybrid 1
Q: I forgot my login, how do I get back in?
keyword D1 D10 D3 | vector D3 D5 D10 | hybrid D3 D1 D10
D3 is at: keyword 3, vector 1, hybrid 1
Q: 150 dollar fee
keyword D12 | vector D8 D1 D12 | hybrid D12 D8 D1
D12 is at: keyword 1, vector 3, hybrid 1
Q: Can I get my money back if I cancel?
keyword D10 D1 D5 | vector D8 D1 D12 | hybrid D1 D10 D5
D1 is at: keyword 2, vector 2, hybrid 1Try: The question is "150 dollar fee" and the ranker starts on Vector, where the stolen-bike passage is only third. Press Search, then switch Rank by to Keyword (BM25) and to Hybrid. Then ask "I forgot my login, how do I get back in?" under all three rankers: which one puts the password passage first, and which puts it third?
Goals
- The stolen-bike passage (the only one with 150) ranks first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Keep passages that share no word
In the Python tab, change keyword_ranking so it keeps every passage, even those with a BM25 score of 0: replace if scores[i] > 0 with if scores[i] >= 0. Run.
Myth vs fact
Myth
Vector search has replaced keyword search.
Fact
Vectors miss exact codes, numbers and rare names (the 150 dollar fee). Most production search systems keep keyword search for exactly those questions.
Myth
Hybrid search is always better than either one.
Fact
It is usually at least as good, but it is not magic: if neither list contains the right passage near the top (the helmet question), merging them cannot help. And it costs more to run. Measure it (Topic 6.7).
Myth
If both rankers agree, the answer is right.
Fact
Agreement is strong evidence, not proof. Two rankers can share a blind spot, such as ignoring the word "not". That is why evaluation uses real questions with known answers.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
The two rankers fail independently in the useful cases, which is why fusing them helps. If they fail together (the same vocabulary gap, a missing document, a bad chunk), no fusion can help. When hybrid disappoints, look at the questions both lists get wrong before adding more machinery.
- ▸
Keep both rankings in the trace: ids, ranks and scores from each list. When an answer is wrong, one glance at "keyword rank 1, vector rank 9" tells you which side to improve (Topic 6.8 returns this trace).
- ▸
Real vector models are better at exact tokens than this toy, and keyword search can be improved with synonyms and stemming (Phase 1). The two lists still differ in *kind* of error, so the case for fusion survives better components. The lab shows both failures with the real model.
Remember this
- 1
Why this exists. Phase 1 built keyword search (BM25) and Phase 3 built vector search. Both return a ranked list for the same question, and both fail silently: the wrong passage comes back looking as confident as the right one. They fail on *different* questions, which is exactly why combining them works. (LLM Engineering Phase 8 sketches RAG in short. This phase goes much deeper on this one step.)
- 2
Where keywords win. A keyword ranker needs the exact word. That is a weakness for paraphrases, but a strength for rare, exact things: error codes, product names, part numbers, amounts. Ask "150 dollar fee". Only one handbook passage (D12, the stolen bike) contains
150, so BM25 ranks it first. A vector model turns the whole question into a general feeling of "money and fees" and the exact number barely matters: in the search lab below the real model puts the monthly pass (cosine about 0.41) and the late fee (about 0.35) above the stolen-bike passage (about 0.21). - 3
Where vectors win. A vector ranker needs only the same *meaning*. Ask "I forgot my login, how do I get back in?". The right passage (D3) talks about resetting a password and shares only the word
forgotwith the question. BM25 puts it third, behind two passages that share small words likeinandget. The real model puts it first with a cosine of about 0.47, far above the next passage (about 0.16). - 4
Where each helps the other. "Can I get my money back if I cancel?" has the refund passage (D1) in second place in both lists (keyword: the flat-tyre passage first, vector: the monthly pass first). Neither ranker would answer correctly. But a passage that sits near the top of *both* lists is a strong candidate, and merging the lists puts D1 first. This is the idea behind hybrid search: agreement between two different views is evidence. Topic 6.3 turns it into a formula.
- 5
Hybrid is not magic. For "What is the age limit for helmets?" the right passage (D4) is in the top 3 of *neither* list, and the hybrid list puts it fifth. Both rankers miss for a similar reason: the question says
age limitandhelmets, the passage saysunder 16andhelmet. Topic 6.4 adds a second stage that re-reads a shortlist. Running two searches also costs more code, more latency and one more thing to monitor, so Topic 6.7 asks when it is worth it. - 6
What the example below shows. Four questions, each ranked three ways: keyword only, vector only, and hybrid. The vector side is a toy embedder built from five hand-made "meaning" lists, so its numbers are not what a real model gives; the lab uses the real one. Read the last line of each block: it says where the right passage landed in each list.
Explain it without notes
Give one example of a question keyword search answers better than vector search, and one the other way round. Explain why.
Why does merging two ranked lists help when each list is sometimes wrong?
What does it mean that both rankers fail silently, and what do you do about it?
Is hybrid search always worth its extra cost?
Practice
Write place(ranking, gold) that returns the 1-based place of gold in a list of ids, or "none" when it is missing. Print it for ["D3", "D1"] with gold D1, and for ["D3"] with gold D8.
Given a keyword ranking ["D2", "D9"] and a vector ranking ["D9", "D4", "D2"], print the ids that appear in both lists, in the order of the vector ranking. Why are such ids good candidates?
Trade-offs
- ↔
Keyword only: cheap, exact on codes and names, blind to paraphrases.
- ↔
Vector only: finds meaning, but can drown an exact number in a general topic.
- ↔
Hybrid: covers both kinds of question, at the cost of two searches, a merge step and more to monitor.
Done when you can
Done when you can name a question keyword search wins and one vector search wins, and say why.
Done when you can explain why a passage high in both lists is a good candidate.
Done when you can read the three places of the right passage in the example's output.
Done when you can name a question that hybrid search still gets wrong, and say what would fix it.