PHASE 1Beginner ~22 min· topic 3 of 9

Topic 1.3

TF-IDF by Hand

In one line

TF-IDF scores a passage by how often each query word appears in it (term frequency), weighted by how rare that word is across all passages (inverse document frequency). It ranks results, and it has two famous flaws.

Think of it like this

Finding who in a room is talking about bicycles. If everyone says "the" all the time, hearing "the" tells you nothing. If only one person says "helmet", and says it three times, you listen to that person. Words that few people use are strong clues; words that everyone uses are noise. TF-IDF is that instinct as arithmetic.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Term frequency (tf): How many times a term appears in one passage.
Document frequency (df): In how many passages the term appears at least once.
IDF: A weight that is high for rare terms and near zero for terms that are everywhere: ln(N / df).
Natural logarithm (ln): The logarithm with base e (about 2.718). It turns big ratios into small, gentle numbers.
Keyword stuffing: Repeating a word many times to win a ranking.
Length bias: Longer passages score higher just because they contain more words.

Step by step

01Step 1: count what is in each passage

Tokenise every passage (Topic 1.1) and keep the word list. tf is words.count(term); df is how many lists contain the term.

counts.pywhole filepython

words_of = {doc_id: tokenize(text) for doc_id, text in HANDBOOK}

def df(term):
    return sum(1 for words in words_of.values() if term in words)

02Step 2: weight words by rarity

idf = ln(N / df). Print it for a few words, as the example does, and see the pattern: the more passages a word is in, the lower its weight. A word in all passages would be exactly 0.

idf.pywhole filepython

import math

def idf(term):
    return math.log(N / df(term))

03Step 3: score and rank

For each passage, add tf * idf for every distinct query term. Terms missing from the whole collection (df = 0) are skipped, since N / 0 is undefined. Sort best first.

04Step 4: read the result

D8 wins with 4.16, D10 is second with 2.08 because of the single word cost, and the rest score 0. The ranking is right here, but look at what carried it: a single rare word.

Step 4: read the resultdiagram

Rendering diagram…

05Step 5: build a passage that cheats

The second example adds a passage that is the word pass eight times. Its score is 5.55, higher than the genuine passage. Raw counts have no ceiling.

06Step 6: build a passage that is just long

Passage C is 45 words with three mentions. It outranks the focused 4-word passage A. Normalising by length (the last column) fixes C but not B, so the formula itself must change.

Try it yourself

1
Predict the idf
In the first example, lock is in 2 of 8 passages. Predict ln(8 / 2) on a calculator, then check the printed value. Then add the word charge to the list and explain why its df is 0 and what happens if you ask for its idf.
2
A new query
Change the query to helmet under 16. Which passage wins and by how much? Why does under carry weight even though it sounds like a small word?
3
Cheat on purpose
In the second example make passage B ten words of pass instead of eight. How does its tf-idf change? Now make A say pass twice. Does the genuine passage ever catch the stuffed one?

Code & diagrams

TF-IDF over eight passages New tab

Natural log in all three languages (math.log, Math.log). Eight handbook passages: D1, D2, D3, D4, D6, D8, D10, D12.

Expected output

idf (N = 8):
  the     df=5  idf=0.470
  a       df=4  idf=0.693
  bike    df=2  idf=1.386
  lock    df=2  idf=1.386
  helmet  df=1  idf=2.079
  monthly df=1  idf=2.079
query: monthly pass cost
  D8  4.159
  D10 2.079
  D1  0.000

The two flaws: repetition and length New tab

Passages A, B and C are made up to expose the flaws. D1 to D3 are handbook passages that make the collection large enough for idf to be above zero.

Expected output

df=3 of 6  idf=0.693
A  tf=1  words=4   tf-idf=0.693  per word=0.173
B  tf=8  words=8   tf-idf=5.545  per word=0.693
C  tf=3  words=45  tf-idf=2.079  per word=0.046

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Turn the weight upside down

In the Python tab of the first example, change math.log(N / df(word)) to math.log(df(word) / N) and run.

terminal

$ Run

── what you'll see ──

idf (N = 8):

the df=5 idf=-0.470

a df=4 idf=-0.693

bike df=2 idf=-1.386

lock df=2 idf=-1.386

helmet df=1 idf=-2.079

monthly df=1 idf=-2.079

query: monthly pass cost

D1 0.000

D2 0.000

D3 0.000

Myth vs fact

Myth

A word that appears more often is always more important.

Fact

Only up to a point. Repetition inside one passage is cheap to fake, and a word that appears in every passage tells you nothing about which one to pick.

Myth

IDF is a property of the word.

Fact

It is a property of the word in your collection. monthly is rare in a bike handbook and common in a banking one, so the same word has different weights in different indexes.

Myth

TF-IDF is what modern search engines use.

Fact

Most moved to BM25, which repairs TF-IDF's flaws. TF-IDF is the simpler stepping stone.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
IDF depends on collection statistics, so it changes as documents are added and removed. Engines that shard an index can compute IDF per shard, which gives slightly different scores for the same document depending on where it lives. Some offer a mode that first collects global statistics.
▸
There are many TF-IDF variants: log-scaled tf (1 + ln(tf)) to soften repetition, smoothed idf (ln(1 + N/df)) so a word in every passage still counts a little, and length normalisation of the vector. They are attempts to patch the two flaws you just saw, which BM25 handles in a principled way.
▸
A term that is in every passage gets IDF 0 here and so can never help or hurt. A query made only of such terms gives all passages score 0 and the order is arbitrary. Your retrieve must handle "nothing scored above zero" explicitly, and Phase 0's refusal rule is how AskDocs does it.

Remember this

1
Why this exists. The index from Topic 1.2 can say *which* passages contain the words, but not which is *best*. Counting shared words (Phase 0) treats the and helmet as equal, and that is why the login question went wrong. Ranking needs two ideas: how much a word shows up in this passage, and how much that word is worth at all.
2
Term frequency (TF) is the number of times a term occurs in a passage. A passage that says pass three times is probably more about passes than one that says it once. In this topic TF is the raw count.
3
Document frequency (DF) is the number of passages that contain the term at least once. Inverse document frequency (IDF) turns that into a weight: idf = ln(N / df), where N is the number of passages and ln is the natural logarithm. A word in every passage gets weight 0; a word in one passage of eight gets about 2.08. In the example, the (in 5 of 8 passages) is worth 0.47, a is worth 0.69, bike and lock 1.39, and helmet and monthly 2.08. Rarity is a property of the collection, not of one passage.
4
The score of a passage is the sum, over the query's terms, of tf * idf. For monthly pass cost over eight handbook passages, D8 scores 4.16 (it has monthly and pass, each worth 2.08), D10 scores 2.08 (it only has cost, in "no extra cost") and every passage without these words scores 0. Notice that costs in D8 does not count for cost: without stemming (Topic 1.1) they are different terms.
5
Flaw one: repetition wins. Passage B in the second example is just the word pass eight times. It scores 5.55, eight times higher than the clean passage A ("Monthly pass: 25 dollars.", 0.69), although it says nothing. Raw counts reward stuffing a word, and early web search engines were exploited this way.
6
Flaw two: long passages win. Passage C is 45 words about many things and mentions pass three times. It scores 2.08 and beats A, which is entirely about the pass. More words means more chances to contain the query. Dividing the count by the passage length (the last column) fixes C, which falls to 0.05, but B is still on top at 0.69. Both flaws need a better formula, and that is BM25 (Topic 1.4).
7
Where TF-IDF is still used. BM25 replaced it as the ranking function in search engines, but TF-IDF remains useful for extracting a passage's most characteristic words and as features for simple machine-learning models. Understanding it is the fastest route to understanding BM25: the same two ingredients, repaired.

Explain it without notes

Define TF, DF and IDF, and explain why IDF is high for rare words.

Why does a repeated word win under raw TF-IDF, and why is that a problem?

Why does a long passage tend to outrank a short focused one, and does dividing by length fully fix it?

What happens to IDF when a term is in every passage, and when it is in none?

Practice

Write idf(n, df) using the natural logarithm and print df=1, df=4 and df=8 for n = 8, each to 3 decimals.

Given three passages, rank them for the query late with raw tf * idf: a = "late fee late", b = "late fee", c = "monthly pass". Print each id with its score to 3 decimals.

Trade-offs

↔
Raw tf: simple and transparent, but rewards stuffing and long passages.
↔
Log-scaled tf or length division: softer, but ad hoc fixes. BM25 does both properly.
↔
A hand-made weighting scheme: fine for learning, but for production use an engine's tuned BM25 (Topic 1.7).

Done when you can

Done when you can compute tf, df and idf for a word in a small collection by hand.
Done when you can explain why the is worth less than helmet.
Done when you can demonstrate keyword stuffing with a three-line example.
Done when you can name TF-IDF's two flaws and what BM25 changes.

Command Palette

Words you'll meet

Step by step

01Step 1: count what is in each passage

02Step 2: weight words by rarity

03Step 3: score and rank

04Step 4: read the result

05Step 5: build a passage that cheats

06Step 6: build a passage that is just long

Try it yourself

Predict the idf

A new query

Cheat on purpose

Code & diagrams

Break it on purpose

Turn the weight upside down

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can