Topic 6.6
Diverse Results
In one line
When the same fact is stored in several places, the top results are copies of one passage and the context window fills with repeats. Maximal marginal relevance trades a little relevance for variety, and a de-duplication step removes true copies.
Think of it like this
You ask three friends where to eat and all three recommend the same pizza place, because they all read the same review. Three recommendations, one idea. If you can only visit one place tonight, you wanted one pizza place and two *different* suggestions: a noodle bar and a curry house.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Redundancy
- How much a passage repeats what the already chosen passages say.
- Maximal marginal relevance (MMR)
- Choosing results one at a time, rewarding relevance and penalising similarity to results already chosen.
- Lambda (lam)
- The MMR dial between relevance (1.0) and variety (0.0).
- Near-duplicate
- A passage that is almost the same as another: a copy with a few words changed.
- Jaccard similarity
- Shared words divided by all different words in both texts: 1.0 means identical word sets.
- Threshold
- The cut-off above which two passages count as copies (here 0.8).
Step by step
01Step 1: see the repeats
Rank by relevance alone with lam = 1.0. The top three are the three refund versions, and the late-fee passage that the question also needs is fourth, outside the top 3.
02Step 2: score relevance and redundancy
For each remaining passage, relevance is the cosine with the question and redundancy is the largest cosine with anything already chosen (0 for the first pick).
relevance = cosine(QUESTION, DOCS[i][1])
redundancy = max((cosine(DOCS[i][1], DOCS[j][1]) for j in chosen), default=0.0)
score = lam * relevance - (1 - lam) * redundancy03Step 3: take the best, repeat
Choose the highest score (the first on a tie), move it from the remaining list to the chosen list, and repeat until you have k. The example prints round by round for lam = 0.7.
04Step 4: try three lambdas
1.0 gives A, B, C. 0.7 gives A, late fee, B. 0.5 gives A, late fee, monthly pass. The relevant late-fee passage appears as soon as variety counts for something.
05Step 5: drop true copies first
Compute Jaccard similarity for every pair and keep a passage only if it is below 0.8 against everything kept before it. Passage 5 is dropped; the paraphrases survive.
06Step 6: choose the order of the steps
De-duplicate before re-ranking (it saves pair scores), and apply MMR at the end, over the few passages you will actually send.
Try it yourself
- 1
Predict lambda
Before running, predict the three picks for
lam = 0.9. Then run it and compare: at what lambda does the late-fee passage first appear? - 2
Add a copy
In the first example add a fourth refund vector
unit([0.88, 0.45, 0.10])and runlam = 0.7withk = 4. Does the fourth copy come before the monthly pass? - 3
Raise the threshold
In the de-duplication example set
THRESHOLD = 0.3. Which passages are dropped now, and was that wanted?
Code & diagrams
The vectors are three-number toys so you can check the arithmetic. Real embeddings have hundreds of numbers; the formula is the same.
Expected output
relevance: refund A 0.901, refund B 0.900, refund C 0.892, late fee 0.852, monthly pass 0.502
lambda 1.0: refund A, refund B, refund C
lambda 0.7: refund A, late fee, refund B
lambda 0.5: refund A, late fee, monthly pass
lambda 0.7 round by round:
round 1: refund A 0.631 refund B 0.630 refund C 0.624 late fee 0.596 monthly pass 0.351
round 2: refund B 0.330 refund C 0.325 late fee 0.429 monthly pass 0.196
round 3: refund B 0.330 refund C 0.325 monthly pass 0.192Expected output
overlap of every pair:
1 1.00 0.39 0.34 0.06 0.87
2 0.39 1.00 0.27 0.03 0.35
3 0.34 0.27 1.00 0.06 0.31
4 0.06 0.03 0.06 1.00 0.08
5 0.87 0.35 0.31 0.08 1.00
passage 5 dropped: a copy of passage 1
kept: 1, 2, 3, 4Try: The question needs two facts (cancelling and late returns), but with Rank by on Vector, the top 3 are all refund passages and the late-fee passage is fourth. Press Search, then solve the challenge in two ways: remove two of the three refund versions with the remove button, or switch the ranker to Hybrid. Then open the grid of every pair's similarity and find the three refund versions.
Goals
- The late-fee passage is in the top 3
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Reward redundancy instead of penalising it
In the Python tab of the first example, in mmr, change lam * relevance - (1 - lam) * redundancy to lam * relevance + (1 - lam) * redundancy and run.
Myth vs fact
Myth
More retrieved passages means more information.
Fact
Five copies of one fact are one piece of information in five times the tokens. Variety, not count, is what extends what the model knows.
Myth
MMR removes duplicates.
Fact
It makes duplicates less likely to be picked together. It does not delete anything from the index, and with lambda close to 1 it barely changes the result.
Myth
Word overlap catches every duplicate.
Fact
It catches copies with small edits. Paraphrases, such as the FAQ and email versions of the refund rule, have low word overlap and need embedding similarity.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
MMR needs passage-to-passage similarity: the candidate vectors, which you already have after the vector stage. It is O(k x n) cosine computations for n candidates, trivial next to the re-ranker.
- ▸
Another common limit: at most m passages per source document or per section, so one long document cannot fill the prompt. This is less principled than MMR but predictable and easy to explain to users.
- ▸
Duplicates also distort evaluation: if the same fact exists in three chunks, a test set with one right id will score a correct retrieval as wrong. Decide how your test set treats copies (Phase 9).
- ▸
Keep a canonical id for copies (a content hash of normalised text) at ingestion (Phase 4) and drop exact copies there. Query-time de-duplication is for near-copies you could not remove at the source.
Remember this
- 1
Why this exists. Search ranks every passage on its own by how well it matches the question. If the corpus holds the same fact three times (a handbook page, an FAQ answer, an email template), all three copies match equally well and fill the top three places. The prompt then spends its budget on repeats, and a second relevant fact (the late fee in our case) is pushed out. The model also tends to treat repetition as emphasis.
- 2
Where copies come from. The same policy in a PDF and on a web page, old and new versions of a document, overlapping chunks (Phase 4 uses overlap on purpose), quoted text in a forwarded email, or two chunks that both contain a shared heading. You will have them. The lab below puts three versions of the refund rule next to the late-fee passage.
- 3
Maximal marginal relevance (MMR). Pick results one at a time. Each round, score every remaining passage as
lam * relevance - (1 - lam) * redundancy, where *relevance* is its similarity to the question and *redundancy* is its highest similarity to anything already picked, then take the best.lam(lambda) is a dial between 1.0 (pure relevance, plain top-k) and 0.0 (pure variety). - 4
By hand, in numbers. In the first example, the question is a vector and five passages have vectors too. Relevance: refund A 0.901, refund B 0.900, refund C 0.892, late fee 0.852, monthly pass 0.502. With
lam = 1.0MMR is plain top 3: the three refund copies. Withlam = 0.7round 1 picks refund A; in round 2 refund B scores 0.7 x 0.900 - 0.3 x (its similarity to A) = 0.330, but the late fee scores 0.429 because it is far less similar to A, so it is picked; the result is A, late fee, B. Withlam = 0.5the third pick is the monthly pass rather than another refund copy. - 5
Choosing lambda. A high value (0.7 to 0.9) keeps the most relevant passages and only breaks up obvious repeats. A low value gives variety at the cost of relevance: the monthly pass is a poor answer to a question about cancelling. There is no universal value. Check on test questions, especially ones that need several facts ("cancel or return late").
- 6
De-duplicating true copies. MMR spreads results; it does not remove copies from the index. A cheap pre-step drops a passage when its word overlap with a better one is very high. The second example uses Jaccard similarity (shared words divided by all words): passage 5 is the first passage plus "See the app." and scores 0.87 against it, so a 0.8 threshold drops it. But the paraphrased copies (passages 2 and 3) score only 0.39 and 0.34 against passage 1: word overlap cannot see paraphrases. For those, use embedding similarity: in the lab, the three refund versions have cosines of about 0.83 to 0.90 to each other, against about 0.52 to 0.55 with the late-fee passage.
- 7
Where it goes. De-duplicate first (cheap, removes true copies, saves re-ranking cost), then fuse and re-rank, then apply MMR or a cap on passages per source document when picking the final few. Diversity is not always wanted: for a question with one right answer, three passages saying it three ways may be a fine safety net. Test it.
Explain it without notes
Why do duplicate passages hurt a RAG answer, even though each one is relevant?
Explain the MMR formula and what lambda does.
Why does word overlap fail to detect a paraphrased duplicate, and what do you use instead?
At which points in the pipeline would you remove duplicates, and why there?
Practice
Write jaccard(a, b) over the sets of lower-case words (re.findall(r"[a-z0-9]+", ...)). Print it for "the bike is red" and "the bike is blue" with 2 decimals.
Write mmr_score(relevance, redundancy, lam) and print the score of a passage with relevance 0.9 and redundancy 0.95 for lam = 0.7 and lam = 0.3, with 3 decimals.
Trade-offs
- ↔
Higher lambda: the most relevant passages, copies may still appear. Lower lambda: more variety, less relevance.
- ↔
Word-overlap de-duplication: fast and exact on copies, blind to paraphrases. Embedding de-duplication: finds paraphrases, needs a tuned threshold.
- ↔
Diversity step: one more thing to tune and test; sometimes redundancy is a useful safety net.
Done when you can
Done when you can compute an MMR round by hand.
Done when you can explain what lambda 1.0 and lambda 0.5 give.
Done when you can name two sources of duplicate passages.
Done when you can say which duplicates word overlap misses.