Command Palette

Search for a command to run...

Hectal
Phase 1Beginner2 of 14 in RAG and Vector Search

Search Before Vectors

Keyword search by hand: terms, the inverted index, TF-IDF and BM25, phrases and fields, real engines, and the baseline every later RAG idea must beat.

Before embeddings, search engines found documents by words. That older idea is still the baseline every modern RAG system is measured against, and it still wins on some questions: exact error codes, product names, rare terms. This phase builds it from nothing, in Python, Java and JavaScript, so that nothing about it stays magic.

You will turn text into terms, build an inverted index (the structure behind every search engine), score passages with TF-IDF and then BM25, and recompute every number by hand. The BM25 you write uses the same formula and constants as the keyword ranker in the search labs, so you can reproduce a lab's scores on paper. You will also see exactly where keyword search fails, which is the reason Phases 2 and 3 exist.

The phase ends by adding keyword search to AskDocs, the product this course builds. It replaces Phase 0's word counting behind the same retrieve interface, and a small test harness measures how much better it is. Then a playbook gives you an analyser, an index and a ranker to copy into any project.

0/9 · 0%
9 topics ~4 h 22 code blocks & diagrams
Start with the first topic
1
1.1

From Text to Terms

Before a search engine can compare a question with a document it turns both into the same clean list of terms: lower-case, no punctuation, maybe without small words and with word endings trimmed. Every one of these choices helps some searches and hurts others.

23 min 2 code practice

2
1.2

The Inverted Index

An inverted index stores, for every term, the list of documents that contain it. A query then reads a few short lists instead of every document, which is why search stays fast as the collection grows.

20 min 1 code practice

3
1.3

TF-IDF by Hand

TF-IDF scores a passage by how often each query word appears in it (term frequency), weighted by how rare that word is across all passages (inverse document frequency). It ranks results, and it has two famous flaws.

22 min 2 code practice

4
1.4

BM25 by Hand

BM25 repairs TF-IDF: repeated words add less and less to the score (saturation, controlled by k1), and long passages are scaled down against the average length (controlled by b). It is the default keyword ranker of search engines.

27 min 3 code practice

5
1.5

Phrases, Fields and Boosts

Word counts ignore order, structure and exactness. Positions let a search match a phrase, fields let a title count for more than the body, and identifiers such as error codes must be matched exactly or not at all.

27 min 4 code practice

6
1.6

Where Keyword Search Wins and Fails

Keyword search finds the words you typed, not the meaning you intended. It misses paraphrases, synonyms and typos, and it is unbeatable on exact identifiers and rare words. Measuring both sides is how you decide what to add next.

27 min 2 code practice

7
1.7

Real Search Engines

You rarely write an inverted index yourself. Lucene-based engines (Elasticsearch, OpenSearch), PostgreSQL full-text search and SQLite FTS5 all implement the ideas of this phase, with analysers, BM25 and fields built in. Know what each gives you, and when your database is enough.

22 min 4 code practice

8
1.8

Build: Keyword Search for AskDocs

AskDocs version 1 swaps Phase 0's word counting for an inverted index with BM25, behind the same retrieve(question, k) interface, and a small harness measures the change.

29 min 2 code practice

9
1.9

Playbook: A Keyword Baseline

Recipes to copy into any project: one analyser for both sides, an index that handles add, update and remove, per-term score explanations, an exact-id lookup, a harness, and the checklist to pass before you add vectors.

21 min 2 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Analyse text into terms

Write tokenize(text): the lower-cased runs of letters and digits, in order.

Beginner
2

Build an inverted index and answer AND queries

docs is a dict of passage id to text, and tokenize(text) is given. Write build_index(docs) returning a dict from term to the list of passage ids that contain it (each id once, in the order of docs).

Beginner
3

Compute a BM25 term score

Write idf(n, df) with BM25's form: ln(1 + (n - df + 0.5) / (df + 0.5)).

Intermediate
4

Find passages that contain a phrase

docs is a dict of passage id to text, and tokenize(text) is given.

Intermediate
5

Look up exact codes before ranking

docs is a dict of passage id to text. A code looks like letters, a hyphen and digits, such as ERR-4012.

Intermediate