Expansion with weights, and when it hurts
Scores are sums of weights: 1.0 for the user's words, 0.5 for added words. The top-3 list for the battery question shows the noise the expansion adds.
Expansion adds words or text to the query so it can match passages written in other words: synonyms and acronyms, or a short guessed answer that is embedded instead of the question. It raises recall, and it can also pull in noise.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write expand(words, table) that returns the words followed by the table's extra words for each of them, without repeats and keeping order. Print the result for ["refund", "cancel"] with table = {"refund": ["refunded", "credit"], "cancel": ["cancelled"]}.
Write score(weights, text) that sums the weights of the words of weights (a dict) found in the text. Print the score of the late-fee sentence for {"late": 1.0, "fee": 0.5, "charged": 0.5}.
Explain it without notes
What problem does query expansion solve, and what is the risk?
Explain HyDE in your own words. Why can a guess retrieve better than the question?
Give a situation where HyDE makes things worse.
How would you stop expansion from lowering precision?
Expected output
Q: refund if I cancel
added words: cancelled cancellations refunded credit
plain top 3: none
expanded top 3: D1 (2.0)
Q: pw help
added words: password reset
plain top 3: none
expanded top 3: D3 (1.0)
Q: how long does the battery charge last
added words: fee charged cost dollars
plain top 3: D7 (2.0)
expanded top 3: D7 (2.0), D6 (1.0), D5 (0.5)