PHASE 1Beginner ~27 min· topic 4 of 9

Topic 1.4

BM25 by Hand

In one line

BM25 repairs TF-IDF: repeated words add less and less to the score (saturation, controlled by k1), and long passages are scaled down against the average length (controlled by b). It is the default keyword ranker of search engines.

Think of it like this

Reading reviews of a restaurant. The first review that says "delicious" convinces you. The tenth one saying "delicious" adds very little. And a one-line review that says "great food" is stronger evidence than the same two words buried in a fifty-line essay about someone's holiday. BM25 builds in both feelings: repetition has diminishing returns, and the same words count for more in a short text.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

BM25: A keyword ranking formula with term-frequency saturation and length normalisation.
k1: The saturation knob. It caps how much repeating a term can add. Common default 1.2.
b: The length-normalisation knob, from 0 (ignore length) to 1 (full). Common default 0.75.
Saturation: Diminishing returns: each extra occurrence adds less than the one before.
Average length (avg_len): The mean number of terms per passage in the collection.
Okapi: The research system where BM25 was developed. The name "Best Matching 25" is its 25th weighting attempt.

Step by step

01Step 1: gather the collection numbers

Compute the number of passages N (12), each passage's length, and the average length (19.417). These are the only global numbers BM25 needs besides each term's df.

02Step 2: idf for each query term

Use ln(1 + (N - df + 0.5) / (df + 0.5)). The example prints the (df 8, idf 0.425), and monthly, pass, cost (df 1, idf 2.159). what and does are not in any passage, so they contribute nothing.

idf.pywhole filepython

def idf(term):
    return math.log(1 + (N - df(term) + 0.5) / (df(term) + 0.5))

03Step 3: the term score

For a term that occurs tf times in a passage of length len, multiply idf by the saturating, length-aware factor. Write it as a function that takes the term and the passage's word list.

term_score.pywhole filepython

def term_score(term, words):
    tf = words.count(term)
    if tf == 0:
        return 0.0
    norm = tf + K1 * (1 - B + B * len(words) / AVG_LEN)
    return idf(term) * tf * (K1 + 1) / norm

Step 3: the term scorediagram

Rendering diagram…

04Step 4: sum over the query's terms

The passage score is the sum of the term scores. Use each distinct query term once (a word repeated in the question is not worth more). Sort best first; break ties by passage order.

05Step 5: open up the winner

The example breaks D8's score into parts: the 0.468, monthly 2.381, pass 2.381. Add them: 5.230, and 5.231 before rounding. Being able to explain a score term by term is the main debugging tool for keyword search.

06Step 6: see saturation and length in isolation

The second example removes idf and the data, and prints only the passage-dependent factor. Read the saturation table down a column (diminishing returns) and the length table along a row (the effect of b).

07Step 7: check against the lab

Run the search lab below with Rank by set to Keyword. Its BM25 column shows 5.23 for D8, 2.43 for D10 and 0.64 for D3, the numbers you just computed. They are the same formula, constants and tokeniser.

Try it yourself

1
Predict a saturation value
Without running anything, compute factor(4, 1.0, 1.2, 0.75) by hand from the formula, then add it to the first table's loop to check. Is it between the values for 3 and 5?
2
Turn the knobs
In the first example change K1, B = 1.2, 0.75 to K1, B = 0.0, 0.75 (no repetition benefit) and then K1, B = 1.2, 0.0 (no length effect). Which scores move, and by how much?
3
Explain a score
Ask the example about D10 instead of D8 (words_of[9]). Which two terms produce its 2.43, and why is the worth more in D10 than in D8?
4
Lab
In the search lab, change a passage so it repeats "pass" many times. Does it overtake the real pass passage under Keyword ranking? Try the same under Vector ranking.

Code & diagrams

BM25 over the twelve handbook passages New tab

Same formula and constants as the search lab's keyword ranker: k1 = 1.2, b = 0.75, idf = ln(1 + (N - df + 0.5) / (df + 0.5)).

Expected output

N=12  average length=19.417 words
  the     df=8  idf=0.425
  monthly df=1  idf=2.159
  pass    df=1  idf=2.159
  cost    df=1  idf=2.159
top 3: D8=5.231, D10=2.430, D3=0.642
D8 has 15 words
  the     tf=1  part=0.468
  monthly tf=1  part=2.381
  pass    tf=1  part=2.381

Saturation and length normalisation, isolated New tab

The factor is the part of the term score that depends on the passage. Multiply it by idf to get the full term score.

Expected output

saturation, a passage of average length
tf   raw     k1=0.5  k1=1.2  k1=3.0
1    1       1.000   1.000   1.000
2    2       1.200   1.375   1.600
3    3       1.286   1.571   2.000
5    5       1.364   1.774   2.500
8    8       1.412   1.913   2.909
20   20      1.463   2.075   3.478
limits: 1.5, 2.2, 4.0 (that is k1 + 1)
length, tf=2 and k1=1.2
len/avg  b=0     b=0.75  b=1
0.5      1.375   1.600   1.692
1.0      1.375   1.375   1.375
2.0      1.375   1.073   1.000
4.0      1.375   0.746   0.647

Reproduce BM25 in the browserreal embeddings

Try: Press Search and look at the BM25 column with Rank by set to Keyword (BM25): D8 should show 5.23, the flat-tyre passage 2.43 and the password passage 0.64, as in the example. Then change the question to just "pass" and predict the new scores before you press Search. Then try "monthly pass monthly pass": a repeated word in the question should not change anything.

QuestionPassages (12)

Rank byTop k (retrieved)

Embedding modelall-MiniLM-L6-v2384 numbers per text. Runs in your browser: the first Search downloads about 23 MB once, and your text never leaves this page.

Goals

The monthly pass passage ranks first

Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Drop the 1 from the idf logarithm

In the Python tab of the first example, change math.log(1 + (N - df(term) + 0.5) / (df(term) + 0.5)) to math.log((N - df(term) + 0.5) / (df(term) + 0.5)) and run.

terminal

$ Run

── what you'll see ──

N=12 average length=19.417 words

the df=8 idf=-0.636

monthly df=1 idf=2.037

pass df=1 idf=2.037

cost df=1 idf=2.037

top 3: D8=3.791, D10=0.990, D1=0.000

D8 has 15 words

monthly tf=1 part=2.381

pass tf=1 part=2.381

Myth vs fact

Myth

BM25 scores are probabilities or percentages.

Fact

They are unbounded positive numbers that depend on the query and the collection. They order results for one query. They say nothing like "87 percent relevant".

Myth

The defaults k1 = 1.2 and b = 0.75 are always best.

Fact

They are good starting points. Short, uniform passages (like FAQ entries) often want a smaller b; tuning them needs an evaluation set (Phase 9).

Myth

BM25 understands the question.

Fact

It only sees which terms match. login and password are as unrelated to it as login and banana. That is the vocabulary mismatch Phases 2 and 3 address.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Some libraries drop the constant (k1 + 1) from the numerator. Every score is multiplied by the same constant, so the ranking is unchanged but the numbers differ. Never compare raw scores between two implementations; compare rankings.
▸
BM25 scores add across terms, so a passage matching many weak terms can beat one matching a single very strong term. For queries with exact identifiers, boost or pre-filter on the identifier (Topic 1.5).
▸
Length normalisation assumes similar-sized passages. If your index mixes tweets and manuals, b cannot be right for both. Chunking (Phase 4) creates passages of controlled size so BM25's assumption roughly holds.
▸
BM25F extends BM25 to several fields (title, body) by combining their term frequencies before saturation, which is usually better than adding two separate BM25 scores as Topic 1.5 does for simplicity.

Remember this

1
Why this exists. Topic 1.3 ended with two flaws: a stuffed passage wins, and a long passage wins. BM25 ("Best Matching 25", from the Okapi research system) was designed to fix both with two tunable knobs. It has been the standard way to rank by keywords for decades, which is why this course uses it as the baseline.
2
The formula. For one query term t and one passage: idf(t) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len / avg_len)). The passage's score is the sum over the query's distinct terms. Here tf is the term's count in the passage, len the passage's word count, avg_len the average over the collection, and k1 = 1.2, b = 0.75 are the common defaults. One number: for tf = 1, len = 15, avg_len = 19.417 and idf = 2.159, the length part is 1 - 0.75 + 0.75 * 15 / 19.417 = 0.829, the bottom is 1 + 1.2 * 0.829 = 1.995, and the score is 2.159 * 1 * 2.2 / 1.995 = 2.38. That is monthly in D8, as you will see below.
3
The IDF form. BM25 uses idf = ln(1 + (N - df + 0.5) / (df + 0.5)). Compared with TF-IDF's ln(N / df) it is smoothed, and the 1 + inside the logarithm keeps it positive for every word. Without it, a word in more than half of the passages would get a negative weight and *lower* a passage's score. The break-it below shows the at -0.64.
4
**Saturation (k1).** Look at the first table. With k1 = 1.2, a term occurring once contributes a factor of 1.000, twice 1.375, five times 1.774, twenty times 2.075. The factor can never exceed k1 + 1 = 2.2. Raw TF-IDF would give 1, 2, 5 and 20. Small k1 saturates quickly (at 0.5 the limit is 1.5), large k1 is nearly linear (at 3.0 twenty mentions give 3.478). So k1 answers: how fast does repetition stop mattering?
5
**Length normalisation (b).** The len / avg_len ratio shrinks the score of long passages and boosts short ones. In the second table (term twice, k1 = 1.2), b = 0 ignores length (1.375 whatever the length), b = 1 scales fully (a passage 4 times the average drops to 0.647), and b = 0.75 sits between (0.746). A passage of exactly average length is unaffected by b.
6
A full example by hand. In the twelve handbook passages the average length is 19.417 words. For What does the monthly pass cost?, monthly, pass and cost each occur in one passage (idf 2.159) and the in eight (idf 0.425); what and does occur nowhere. D8 has 15 words, shorter than average. Its monthly and pass each contribute 2.381 and the 0.468, which add up to 5.231. D10 is next with 2.430 (it has cost and the) and D3 third with 0.642 (only the). The search lab on this page prints the same numbers, which is the point: there is no hidden magic in the keyword column.
7
What the number means. A BM25 score is not a probability and not a percentage. It depends on the query (a query with more or rarer words scores higher) and on the collection, so scores from different questions cannot be compared: D2 scores 10.39 for the holidays question while D8 scores 5.23 for the pass question, and both are right answers. Use scores to order passages for one query, and be careful with fixed thresholds across queries (Topic 8.4).

Explain it without notes

Write the BM25 score for one term and explain each part.

What does k1 control, and what is the maximum value of the saturating factor?

What does b control, and what do b = 0 and b = 1 mean?

Why does the 1 + inside BM25's idf matter?

Why can't you compare BM25 scores between two different queries?

Practice

Write idf(n, df) with BM25's form and print it for n = 12 and df = 1, 8 and 12, to 3 decimals.

Write factor(tf, k1=1.2) for a passage of average length (len = avg_len, so the length part is 1) and print it for tf = 1, 2, 5.

Trade-offs

↔
Small k1: repetition stops mattering quickly, good against stuffing, weaker for long documents where frequency is informative.
↔
Large b: strongly favours short passages. Small b: lets long passages win. Tune it on your own questions.
↔
BM25 vs vectors: BM25 is exact, explainable and cheap; vectors understand paraphrase and cost more. Phase 6 combines them.

Done when you can

Done when you can compute one term's BM25 contribution from the formula with a calculator.
Done when you can reproduce the lab's keyword scores for the monthly pass question.
Done when you can explain k1 and b with a table of numbers, not only words.
Done when you can say why BM25 scores from different queries cannot be compared.

Command Palette

Words you'll meet

Step by step

01Step 1: gather the collection numbers

02Step 2: idf for each query term

03Step 3: the term score

04Step 4: sum over the query's terms

05Step 5: open up the winner

06Step 6: see saturation and length in isolation

07Step 7: check against the lab

Try it yourself

Predict a saturation value

Turn the knobs

Explain a score

Lab

Code & diagrams

Break it on purpose

Drop the 1 from the idf logarithm

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can