A keyword baseline on ten questions
Plain BM25 over the twelve handbook passages, no threshold. Phase 9 builds a proper evaluation harness; this one is only large enough to see the pattern.
Keyword search finds the words you typed, not the meaning you intended. It misses paraphrases, synonyms and typos, and it is unbeatable on exact identifiers and rare words. Measuring both sides is how you decide what to add next.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write unmatched_words(question, passages, stop) returning the question's words (in order, no repeats) that are not in stop and appear in none of the passages. Test it with the question "Where do lost umbrellas go?", the passages "Items left on a bike are kept at the nearest shop for 14 days." and "A helmet is included free with every rental.", and stop = {"where", "do", "go"}.
Write looks_like_id(question) that is true when the question contains a code shaped like letters, a hyphen and digits (ERR-4012). Print it for three questions.
Explain it without notes
Name three kinds of question keyword search handles badly and one it handles very well.
Why did the money-back question retrieve the flat-tyre passage first?
Why can a high hit rate at 3 on a 12-passage collection be misleading?
Why measure keyword search before adding vector search?
Expected output
want got at 1 in top 3 question
D8 D8 yes yes What does the monthly pass cost?
D3 D1 NO yes I forgot my login, how do I get back in?
D1 D10 NO yes Can I get my money back if I cancel?
D6 D6 yes yes How much do I pay when I bring the bike back late?
D10 D10 yes yes My tyre is flat, what now?
D7 D7 yes yes How far can an electric bike go?
D9 - NO NO Where do lost umbrellas go?
D12 D10 NO yes What if somebody steals my bike?
D11 D11 yes yes How old do I need to be?
D4 D4 yes yes Is a helmet provided?
hit rate at 1: 0.60
hit rate at 3: 0.90