Topic 1.6
Where Keyword Search Wins and Fails
In one line
Keyword search finds the words you typed, not the meaning you intended. It misses paraphrases, synonyms and typos, and it is unbeatable on exact identifiers and rare words. Measuring both sides is how you decide what to add next.
Think of it like this
A very literal friend who helps you search a pile of papers by looking only at exact words. Ask her "find the paper about my head when cycling" and she finds nothing, because the paper says "helmet". Ask her for "invoice 48213" and she finds it instantly and never gets it wrong. She is not stupid: she is literal. Keyword search is that friend.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Vocabulary mismatch
- The question and the answer use different words for the same meaning.
- Synonym
- A different word with (nearly) the same meaning, like refund and money back.
- Typo tolerance
- Matching words that are a few characters wrong (edit distance).
- Baseline
- The simple method every new method is measured against.
- Hit rate at k
- The share of test questions whose right passage is in the top k results.
- Explainability
- Being able to say why a result was returned. With keywords: these words matched.
Step by step
01Step 1: write a test set first
Ten questions, each with the one handbook passage that answers it. Include easy ones (keyword-friendly), paraphrases (login), a missing-vocabulary question (umbrellas) and a word-form question (steals). A test set is what turns an opinion into a measurement (Phase 9).
TESTS = [
("What does the monthly pass cost?", "D8"),
("I forgot my login, how do I get back in?", "D3"),
("Where do lost umbrellas go?", "D9"),
]02Step 2: run the BM25 you built
For each question compute scores for all twelve passages, keep the three best with a score above 0, and compare with the expected id.
03Step 3: read the table
Easy rows say yes yes. The login row shows D1 first and D3 in the top three: found, but behind a wrong passage. The money-back row shows D10 first. The umbrella row shows - and NO NO: nothing matched at all.
04Step 4: see why the wrong one won
For "Can I get my money back if I cancel?" the winner is the flat-tyre passage D10 (BM25 about 2.90) because it contains both get and if; the refund passage D1 (about 1.60) contains only get. Neither word has anything to do with refunds. Printing per-term scores (Topic 1.4) shows this in one glance.
05Step 5: try the same question with real embeddings
In the search lab, the vector ranker puts D1 first with a cosine of about 0.53 and the next passage at about 0.33. The same question that confused BM25 is easy for a model that compares meaning. The lab starts on Keyword so the goal fails; switch the ranker to Vector to solve it.
06Step 6: remember the other side
The same lab with an error code (Topic 1.5) goes the other way: keyword search finds ERR-4107 and the vector ranker does not. A good system uses both kinds of evidence. That is Phase 6's subject.
Try it yourself
- 1
Add a question
Add
("Do you have helmets for kids?", "D4")to the tests. Predict whether BM25 gets it at 1 before running. - 2
Stem the harness
Copy the
stemfunction from Topic 1.1 and apply it when you tokenise both the passages and the questions. Which rows change? Doesstealsnow findstolen? - 3
Find the cause
For each
NOin the table, decide: is it a synonym, a word-form problem, or missing vocabulary? Which of these could stemming fix, and which can only meaning fix? - 4
Compare in the lab
Run every one of the ten questions in the lab under Keyword and under Vector. For which does the first result differ? Make a list of questions where each ranker wins.
Code & diagrams
Plain BM25 over the twelve handbook passages, no threshold. Phase 9 builds a proper evaluation harness; this one is only large enough to see the pattern.
Expected output
want got at 1 in top 3 question
D8 D8 yes yes What does the monthly pass cost?
D3 D1 NO yes I forgot my login, how do I get back in?
D1 D10 NO yes Can I get my money back if I cancel?
D6 D6 yes yes How much do I pay when I bring the bike back late?
D10 D10 yes yes My tyre is flat, what now?
D7 D7 yes yes How far can an electric bike go?
D9 - NO NO Where do lost umbrellas go?
D12 D10 NO yes What if somebody steals my bike?
D11 D11 yes yes How old do I need to be?
D4 D4 yes yes Is a helmet provided?
hit rate at 1: 0.60
hit rate at 3: 0.90Try: The lab starts with the keyword ranker, and the refund passage is not first, so the goal fails. Read the BM25 column to see which passage wins and why. Then switch Rank by to Vector and search again. Then try "My tire is flat" and "What does the monthly pass cost?" with each ranker: which questions does each ranker find easy?
Goals
- The refund passage ranks first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to lower-case
In the Python tab, in the tokenizer, change re.findall(r"[a-z0-9]+", text.lower()) to re.findall(r"[a-z0-9]+", text) (no lower-casing) and run.
Myth vs fact
Myth
Vector search makes keyword search obsolete.
Fact
Vectors are weak on exact codes, rare names and brand-new terms, which keyword search handles perfectly. Strong systems use both (Phase 6).
Myth
If keyword search finds the right passage in the top 3, retrieval is fine.
Fact
In a 12-passage collection, top 3 is a quarter of everything. Judge on a collection and a test set that look like your real ones.
Myth
Adding synonyms by hand solves the vocabulary problem.
Fact
A synonym list helps for the cases you thought of and costs upkeep forever. It is a reasonable patch (Topic 7.3), not a general solution.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Make the evaluation set a living file: every failed real question becomes a row. After a few weeks it describes your users better than any benchmark, and it is the input to Phase 9.
- ▸
Report hit rate at k for the k you actually send to the model, and also hit rate at 1 when you show only the top source to a user. They answer different questions.
- ▸
Think about *cost of failure*. If wrong answers are costly (medical, legal), keyword search's explainability and exactness are valuable; if recall on casual questions is what matters (support chat), meaning-based search pays off more.
Remember this
- 1
Why this topic exists. Before adding anything more complex, you should know exactly what keyword search gets wrong and what it gets right. Otherwise you cannot tell whether embeddings (Phase 2) and hybrid search (Phase 6) are worth their cost, and you cannot defend the choice in a design review.
- 2
Failure 1: synonyms and paraphrases. The user's words differ from the document's:
loginandpassword,money backandrefund,stealsandstolen. The question "I forgot my login, how do I get back in?" shares onlyforgotwith the password passage. In the harness below, the password passage D3 is not first, and for the money-back question the right passage D1 is also not first. - 3
Failure 2: word forms, typos and spelling variants. Without stemming,
stealsdoes not matchstolen, and without fuzzy matchingtiredoes not matchtyre. Stemming (Topic 1.1) and typo tolerance (edit distance) repair some of this and add their own mistakes. - 4
Failure 3: other languages and missing vocabulary. A question in another language shares no words with an English passage, and a question such as "Where do lost umbrellas go?" has no passage with
umbrellasin it at all: the harness row for D9 shows-, nothing was retrieved. A model that knowsumbrellais something that gets left on a bike can do better; counting words cannot. - 5
Where keyword search wins. Exact identifiers and codes (Topic 1.5). Rare technical words, product names and surnames. Words a model has never seen (a product launched yesterday has no embedding knowledge but is a perfect keyword). Cases where you need to explain why something was returned: the matching words are the explanation. And cost: no model, no GPU, no vector storage, milliseconds.
- 6
Reading the harness. The example runs ten handbook questions through the BM25 you built, each with the passage that should answer it. The right passage is first for 6 of 10 (hit rate at 1 is 0.60) and in the top three for 9 of 10 (0.90). Resist the cheerful reading: the handbook has only twelve passages, so "top three" is a quarter of everything. With twelve thousand passages the same failures would push the right answer far out of sight.
- 7
Why it stays the baseline. Even after you add vectors, BM25 remains the thing to beat, and also part of the answer: hybrid systems (Phase 6) run both. Published comparisons across many domains have found BM25 a strong baseline that neural retrievers do not always beat, especially outside the data they were trained on. Always measure a new retriever against it on your own questions.
Explain it without notes
Name three kinds of question keyword search handles badly and one it handles very well.
Why did the money-back question retrieve the flat-tyre passage first?
Why can a high hit rate at 3 on a 12-passage collection be misleading?
Why measure keyword search before adding vector search?
Practice
Write unmatched_words(question, passages, stop) returning the question's words (in order, no repeats) that are not in stop and appear in none of the passages. Test it with the question "Where do lost umbrellas go?", the passages "Items left on a bike are kept at the nearest shop for 14 days." and "A helmet is included free with every rental.", and stop = {"where", "do", "go"}.
Write looks_like_id(question) that is true when the question contains a code shaped like letters, a hyphen and digits (ERR-4012). Print it for three questions.
Trade-offs
- ↔
Keyword only: cheap, explainable and exact, but literal.
- ↔
Vectors only: understand meaning, but weak on exact codes and rare names, and need a model and storage.
- ↔
Both (hybrid): the best of each and more to run, which Phase 6 shows how to measure.
Done when you can
Done when you can name the three failure families of keyword search with an example of each from the handbook.
Done when you can name five things keyword search does better than vectors.
Done when you can run a small test set and report hit rate at 1 and at k.
Done when you can explain why a tiny collection flatters recall.