The two flaws: repetition and length
Passages A, B and C are made up to expose the flaws. D1 to D3 are handbook passages that make the collection large enough for idf to be above zero.
TF-IDF scores a passage by how often each query word appears in it (term frequency), weighted by how rare that word is across all passages (inverse document frequency). It ranks results, and it has two famous flaws.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write idf(n, df) using the natural logarithm and print df=1, df=4 and df=8 for n = 8, each to 3 decimals.
Given three passages, rank them for the query late with raw tf * idf: a = "late fee late", b = "late fee", c = "monthly pass". Print each id with its score to 3 decimals.
Explain it without notes
Define TF, DF and IDF, and explain why IDF is high for rare words.
Why does a repeated word win under raw TF-IDF, and why is that a problem?
Why does a long passage tend to outrank a short focused one, and does dividing by length fully fix it?
What happens to IDF when a term is in every passage, and when it is in none?
Expected output
df=3 of 6 idf=0.693
A tf=1 words=4 tf-idf=0.693 per word=0.173
B tf=8 words=8 tf-idf=5.545 per word=0.693
C tf=3 words=45 tf-idf=2.079 per word=0.046