Topic 6.2
Mixing Scores (and Why It Is Hard)
In one line
The obvious way to combine BM25 and cosine is to add the scores. They live on different scales, so you must normalise first, and even then the weights you tuned stop working when the data changes.
Think of it like this
A school wants one ranking from a maths test marked out of 100 and a drawing test marked out of 5. Adding the marks gives the maths test twenty times the say. Converting both to a percentage first helps, but if one new pupil scores 500 on a bonus maths paper, every other pupil's maths percentage shrinks, and the order changes although nothing about them changed.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Score scale
- The range of values a scoring method produces. BM25 is open-ended, cosine stays between -1 and 1.
- Normalise
- Rescale scores to a common range so different methods can be compared or combined.
- Min-max normalisation
- Rescale a list to 0..1 using its own lowest and highest value: (score - lowest) / (highest - lowest).
- Weighted sum
- Add several scores, each multiplied by a weight that says how much to trust it.
- Alpha
- The weight of the keyword score in the mix. The vector score gets 1 - alpha.
- Outlier
- A value far outside the others, such as one very high BM25 score.
Step by step
01Step 1: look at the raw scores
Print BM25 and cosine for every passage for one question. The first column is open-ended, the second is capped. Putting them next to each other shows why adding them would let the larger number decide.
02Step 2: rescale each list to 0..1
Write min_max(scores). The best score becomes 1, the worst 0, everything else falls between. If all scores are equal there is nothing to rescale: return zeros rather than dividing by zero.
def min_max(scores):
low, high = min(scores), max(scores)
return [0.0 if high == low else (s - low) / (high - low) for s in scores]03Step 3: mix with a weight
alpha * keyword + (1 - alpha) * vector per passage, then sort. With alpha = 0.5 both lists count equally. The function returns the top 3 with their mixed scores.
04Step 4: change the weight
Run the same question with alpha = 0.2. The winner changes from D12 (correct) to D8. A single number you chose decided the answer, and you only know the right value if you have test questions.
05Step 5: change the data
Add D15, a price list that repeats "150 dollar fee" four times. Its BM25 score is the new maximum, so every other passage's normalised keyword score drops. D15 wins and D12's lead shrinks from 0.298 to 0.097. The ranking of old passages changed with no change to the passages themselves.
06Step 6: why this leads to ranks
Both problems come from using score *values*. A rank (first, second, third) is the same whether BM25 is 5 or 50, and one outlier changes ranks by at most one place. Topic 6.3 builds the method that uses only ranks.
Try it yourself
- 1
Find a good alpha
In the second example, try
alphavalues 0.9, 0.7, 0.3 and 0.1 for the "150 dollar fee" question. For which range of alpha is D12 first? Is the same range right for the login question? - 2
Predict the outlier
Change D15 so it repeats the phrase only once. Predict whether D15 still wins at
alpha = 0.5, then run. - 3
Skip the normalising
Replace
min_max(...)by the raw lists inmixed_topand run atalpha = 0.5. Which list decides the order?
Code & diagrams
BM25 here uses the same formula and constants as the search lab (k1 = 1.2, b = 0.75). The cosine column is the toy embedder's.
Expected output
passage bm25 cosine
D1 0.000 0.743
D2 0.000 0.000
D3 0.642 0.000
D4 0.000 0.000
D5 0.563 0.316
D6 0.458 0.436
D7 0.000 0.000
D8 5.231 0.816
D9 0.480 0.000
D10 2.430 0.229
D11 0.438 0.000
D12 0.636 0.487
bm25 runs from 0.000 to 5.231; cosine from 0.000 to 0.816
best passage: bm25 says D8, cosine says D8D15 is a passage added to the handbook for this example only. The toy embedder knows none of its words, so its vector score is 0.
Expected output
150 dollar fee
alpha 0.5: D12=0.798, D8=0.500, D1=0.455
alpha 0.2: D8=0.800, D1=0.728, D12=0.677
alpha 0.5 + D15: D15=1.000, D12=0.505, D8=0.408
I forgot my login, how do I get back in?
alpha 0.5: D3=0.883, D10=0.572, D1=0.500
alpha 0.2: D3=0.953, D10=0.371, D5=0.338
alpha 0.5 + D15: D3=0.875, D10=0.572, D1=0.500Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Add raw scores without normalising
In the Python tab of the second example, change the keyword line of mixed_top to use raw BM25: keyword = bm25_scores(texts, question) (without min_max). Run.
Myth vs fact
Myth
A hybrid score is just keyword score plus vector score.
Fact
The two scores are not in the same units. Added raw, the bigger scale wins. Even normalised, the result depends on one weight and on the extremes of each result list.
Myth
Once I tune alpha, I am done.
Fact
Alpha is tuned for one corpus and one kind of question. A change of embedding model, new documents or a different kind of user query can make it wrong again.
Myth
Cosine of 0.8 means 80% relevant.
Fact
A cosine is not a probability. Its meaning depends on the model: some models give 0.3 to good matches and others 0.8. Compare scores only within one model.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Min-max uses the *current result list's* extremes, so it is unstable per query: the best passage always gets 1, even when it is a poor match. A weak best match and a strong best match both look like 1.0. Absolute meaning is lost, so you cannot use a normalised score as an evidence threshold (Topic 8.4).
- ▸
If you keep scores, normalise with statistics from a fixed sample of past queries (for example the 95th percentile of BM25 for your corpus) instead of the current list. That is more stable but needs maintenance and still drifts as the corpus grows.
- ▸
Some databases offer a weighted hybrid directly. Whatever you use, the same two questions apply: how are the scales aligned, and who re-tunes the weight when the data changes? RRF (next topic) is popular partly because it removes both questions.
Remember this
- 1
Why this exists. The first idea when you have two scores for each passage is to add them. It fails because the two scores are not measured in the same units. BM25 is open-ended: it grows with how rare and frequent the matched words are, and it can be 5 or 12 or 30. Cosine similarity between embeddings stays between -1 and 1, and for a text model it is usually somewhere near 0.0 to 0.8. Adding a BM25 of 5.2 to a cosine of 0.8 means the keyword score decides everything.
- 2
The scales, side by side. In the first example below, for "What does the monthly pass cost?", BM25 runs from 0.000 to 5.231 and the toy cosine from 0.000 to 0.816. Even with this small gap the number that dominates is the larger one. And the scales also *move*: BM25 depends on the whole corpus (document frequencies and average length), cosine on the model, so neither range is fixed for a given product.
- 3
Normalising. The usual fix is min-max normalisation: rescale each list to 0..1 with
(score - lowest) / (highest - lowest), so the best passage in each list gets 1 and the worst gets 0. For example, the scores 2, 4 and 6 become (2-2)/4 = 0, (4-2)/4 = 0.5 and (6-2)/4 = 1. Then take a weighted sum:alpha * keyword + (1 - alpha) * vector, wherealphasays how much you trust the keyword list. Withalpha = 0.5on "150 dollar fee", the stolen-bike passage D12 scores 0.798 and wins; the monthly pass D8 scores 0.500. - 4
Weights are fragile. Change
alphato 0.2 (trust vectors more) and the same question gives D8 (0.800) first and D12 only third (0.677). So the weight decides the winner, and the right value depends on your questions: a catalogue of part numbers wants a high alpha, a help centre of free text wants a low one. - 5
Normalisation is fragile too. Min-max uses the highest and lowest score *in this one result list*. Add one passage that repeats the phrase (a price list, call it D15) and the highest BM25 score jumps; every other passage's normalised keyword score shrinks. In the example D12 falls from 0.798 to 0.505 and D15 takes first place, with D12's margin over D8 dropping from 0.298 to 0.097. Nothing about D12 or D8 changed, yet their scores did. One unusual document can reshuffle the order of unrelated ones.
- 6
What people do about it. Other normalisations exist (z-scores, rank-based), and some systems learn the weights from labelled data. All need tuning and re-tuning as the corpus changes. The next topic shows a method that avoids the scale problem completely by throwing the scores away and keeping only the ranks.
Explain it without notes
Why can't you simply add a BM25 score and a cosine similarity?
How does min-max normalisation work, and what is one weakness?
Explain how adding a single document can change the order of two unrelated documents in a weighted-sum hybrid.
Why can't a min-max normalised score be used to decide that no passage is good enough?
Practice
Write mix(keyword, vector, alpha) that takes two lists of scores already scaled to 0..1 and returns the list of mixed scores. Print the result for [1.0, 0.0], [0.0, 1.0] and alpha = 0.7.
Write min_max(scores) that returns zeros when all scores are equal. Print it for [2, 4, 6] and for [3, 3].
Trade-offs
- ↔
Weighted sum: simple and tunable, but needs normalising and a weight that is re-tuned as data changes.
- ↔
Normalising per result list: easy, but unstable and loses absolute meaning.
- ↔
Normalising with fixed corpus statistics: stable, but needs maintenance.
Done when you can
Done when you can explain why raw BM25 and cosine cannot be added.
Done when you can write min-max normalisation, including the all-equal case.
Done when you can show with the example how the weight changes the winner.
Done when you can explain how one new document can reshuffle unrelated scores.