Dropping near-copies by word overlap
When the same fact is stored in several places, the top results are copies of one passage and the context window fills with repeats. Maximal marginal relevance trades a little relevance for variety, and a de-duplication step removes true copies.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write jaccard(a, b) over the sets of lower-case words (re.findall(r"[a-z0-9]+", ...)). Print it for "the bike is red" and "the bike is blue" with 2 decimals.
Write mmr_score(relevance, redundancy, lam) and print the score of a passage with relevance 0.9 and redundancy 0.95 for lam = 0.7 and lam = 0.3, with 3 decimals.
Explain it without notes
Why do duplicate passages hurt a RAG answer, even though each one is relevant?
Explain the MMR formula and what lambda does.
Why does word overlap fail to detect a paraphrased duplicate, and what do you use instead?
At which points in the pipeline would you remove duplicates, and why there?
Expected output
overlap of every pair:
1 1.00 0.39 0.34 0.06 0.87
2 0.39 1.00 0.27 0.03 0.35
3 0.34 0.27 1.00 0.06 0.31
4 0.06 0.03 0.06 1.00 0.08
5 0.87 0.35 0.31 0.08 1.00
passage 5 dropped: a copy of passage 1
kept: 1, 2, 3, 4