TF-IDF over eight passages
Natural log in all three languages (math.log, Math.log). Eight handbook passages: D1, D2, D3, D4, D6, D8, D10, D12.
TF-IDF scores a passage by how often each query word appears in it (term frequency), weighted by how rare that word is across all passages (inverse document frequency). It ranks results, and it has two famous flaws.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write idf(n, df) using the natural logarithm and print df=1, df=4 and df=8 for n = 8, each to 3 decimals.
Given three passages, rank them for the query late with raw tf * idf: a = "late fee late", b = "late fee", c = "monthly pass". Print each id with its score to 3 decimals.
Explain it without notes
Define TF, DF and IDF, and explain why IDF is high for rare words.
Why does a repeated word win under raw TF-IDF, and why is that a problem?
Why does a long passage tend to outrank a short focused one, and does dividing by length fully fix it?
What happens to IDF when a term is in every passage, and when it is in none?
Expected output
idf (N = 8):
the df=5 idf=0.470
a df=4 idf=0.693
bike df=2 idf=1.386
lock df=2 idf=1.386
helmet df=1 idf=2.079
monthly df=1 idf=2.079
query: monthly pass cost
D8 4.159
D10 2.079
D1 0.000