PHASE 1Beginner ~21 min· topic 9 of 9

Topic 1.9

Playbook: A Keyword Baseline

In one line

Recipes to copy into any project: one analyser for both sides, an index that handles add, update and remove, per-term score explanations, an exact-id lookup, a harness, and the checklist to pass before you add vectors.

Think of it like this

A chef's mise en place. Before service, every ingredient is washed, cut and placed within reach, so cooking is fast and mistakes are rare. These recipes are your mise en place for search: prepared once, reused in every project.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Upsert: Insert a record, or replace it if it already exists.
Stale term: A term still in the index although the document no longer contains it.
Analyser version: A label saved with the index so you know which analysis built it.
Baseline: The simple method whose numbers every later change must beat.
Miss list: The ids of test passages that were not retrieved: your to-do list.

Step by step

01Step 1: one analyser function

The example's analyze lower-cases, splits, drops stop words and stems. The KeywordIndex receives it as an argument and calls it for documents and for queries.

02Step 2: postings that can change

add removes any old version first, counts the terms, stores the passage length, and updates the postings. remove deletes the passage from every list and drops lists that become empty.

03Step 3: search with statistics from the current index

N, df and the average length come from the index as it is now, so scores change when documents change. After the update the new D8 scores 2.215 for the question about the pass cost; with no D8 at all the result is (none).

Step 3: search with statistics from the current indexdiagram

Rendering diagram…

04Step 4: stemming in action

The first search, charges for late bikes, finds D6 and D9 even though D6 says charged and bikes, because the same stemmer ran on both sides. D9 appears because bike matches; it is a weak hit (0.512).

05Step 5: where the real engine plugs in

The polyShow block replaces KeywordIndex.search with a call to Elasticsearch over plain HTTP. The caller keeps the same retrieve(question, k) shape; only the inside changes.

06Step 6: run the checklist

Before you add vectors, tick every item in the checklist below. If one fails, fix it first: vectors will not repair broken analysis, and a missing baseline means you will not know whether they helped.

Try it yourself

1
Update, not just add
Call index.add("D9", "Items left on a bike are kept for 30 days at the nearest shop.") and search for kept. Print len(index.postings) before and after. Which terms appear or vanish?
2
Different analyser
Build a second index with an analyser that does not stem. Run charges for late bikes on both. Which finds D6, and why?
3
Explain a hit
Copy the explain idea from Topic 1.8's practice into KeywordIndex and print the parts for D8 and pass cost.

Code & diagrams

Recipes: one analyser and an index you can update New tab

Production-shaped but small: incremental postings, one analyser object, ties broken by id. A real engine adds persistence, segments and concurrency.

Expected output

documents: 3, terms: 27
'charges for late bikes': D6=1.394, D9=0.512
'what does a pass cost':  D8=1.885
after updating D8:        D8=2.215
documents: 3, terms: 23
after removing D8:        (none)
documents: 2, terms: 18

Where a real engine plugs in: Elasticsearch over HTTPpython

Shown, not run. The index must already exist (Topic 1.7). The same request works against OpenSearch. In Java, parse the JSON with Jackson.

import os

import httpx

ES_URL = os.environ["ES_URL"]  # for example https://search.example.com
ES_API_KEY = os.environ["ES_API_KEY"]


def retrieve(question, k=3):
    """Same shape as the in-memory retriever: scored hits, best first."""
    body = {"size": k, "query": {"multi_match": {"query": question, "fields": ["title^3", "text"]}}}
    reply = httpx.post(
        ES_URL + "/handbook/_search",
        json=body,
        headers={"Authorization": "ApiKey " + ES_API_KEY},
        timeout=10,
    )
    reply.raise_for_status()
    return [(hit["_score"], hit["_source"]) for hit in reply.json()["hits"]["hits"]]

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Update without removing the old version

In the Python tab, delete the line self.remove(doc_id) at the start of add and run.

terminal

$ Run

── what you'll see ──

documents: 3, terms: 27

'charges for late bikes': D6=1.394, D9=0.512

'what does a pass cost': D8=1.885

after updating D8: D8=2.928

documents: 3, terms: 29

after removing D8: (none)

documents: 2, terms: 18

Myth vs fact

Myth

A baseline is a throwaway step.

Fact

It is the measuring stick, and in production it often remains the keyword half of a hybrid system (Phase 6).

Myth

If the index is rebuilt nightly, updates do not need care.

Fact

Until the rebuild, deleted documents stay searchable. That matters when the deletion is a privacy or legal request (Topic 10.6).

Myth

Stemming and stop words are settings you can flip later.

Fact

They change every stored term. Changing them means re-indexing everything.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Write an index-equals-rebuild test: after a random series of add, update and remove calls, the index must have exactly the same postings as one built from the final set of documents. It finds every bug of the kind shown in the break-it.
▸
Log the analysed query terms and the top hits with their parts for a sample of real traffic. Most relevance bugs are visible there at a glance.
▸
Keep the exact-id lookup and the BM25 path as two stages with a clear order. The first returns exact matches or nothing; the second ranks. A fusion of the two arrives in Phase 6.

Remember this

1
Recipe: one analyser. When to use it: always. What it guards against: the index and the query being analysed differently (Topic 1.1), the commonest silent failure in keyword search. How to adapt it: write analyze(text) once, store its version next to the index, and pass the same function to the index and to the query path. Change it only together with a re-index.
2
Recipe: an index that can change. When to use it: whenever documents are added, edited or deleted. What it guards against: stale terms. An update must remove the old postings first, or old words stay searchable and counts are doubled; the break-it shows the inflated score and the stale terms. How to adapt it: in a database or engine this is upsert and delete; the habit is to treat an edit as remove then add.
3
Recipe: explain every score. When to use it: in tests, in logs and when someone says "search is wrong". What it guards against: arguing from impressions. Return per-term parts (Topic 1.4) for any hit, and log the question's terms after analysis. A wrong ranking is nearly always visible in those few numbers.
4
Recipe: exact id first. When to use it: when questions can contain error codes, order numbers, part numbers or names. What it guards against: confident wrong answers for mistyped ids (Topic 1.5). How to adapt it: keep identifiers in an unanalysed field, detect the id shape with a regular expression, look up exactly, and answer "no such code" when there is no match.
5
Recipe: the harness. When to use it: before and after every change. What it guards against: improvements that are really regressions. Keep a file of questions with the passage ids that answer them, report hit rate at 1 and at k, and list the misses by id (Topic 1.8). Add every real failure to the file.
6
Recipe: choose the engine last. When to use it: when moving to production. Prototype with the in-memory index, then pick PostgreSQL or a dedicated engine (Topic 1.7) and run the same harness against it. If the numbers match, switch; if not, inspect the analysis.

Explain it without notes

Why must an update remove the old postings before adding the new ones?

What is the single most important rule about the analyser, and how do the recipes enforce it?

Why check an identifier with an exact match before ranking?

Practice

Write missing_terms(index_terms, text), a tiny check that returns the list of terms in text (lower-cased runs of letters and digits) that are missing from the set index_terms. Print it for the index terms {"monthly", "pass"} and the text "The monthly pass".

Write rebuild_equals(postings_a, postings_b) comparing two dicts of term -> {doc: tf} and returning "same" or "different". Print the result for two small dicts that are equal and two that differ in one count.

Trade-offs

↔
Incremental updates: fresh results immediately, more code and more places for bugs than nightly rebuilds.
↔
A strict exact-id stage: honest answers for codes, but no help with typos unless you add a fuzzy suggestion.
↔
In-memory prototype vs engine: faster to learn and test, then a migration to plan.

Done when you can

Before you add vectors: one analyze function is used for documents and queries, and its version is stored with the index.
Before you add vectors: adding, updating and removing a document gives the same index as rebuilding from scratch.
Before you add vectors: you can print the analysed query terms and per-term score parts for any hit.
Before you add vectors: exact identifiers are matched exactly, and a wrong code says so.
Before you add vectors: a test file of real questions gives you hit rate at 1 and at k, and a list of misses.
Before you add vectors: you know which failures are vocabulary mismatches, so you can judge whether embeddings fixed them.

Command Palette

Words you'll meet

Step by step

01Step 1: one analyser function

02Step 2: postings that can change

03Step 3: search with statistics from the current index

04Step 4: stemming in action

05Step 5: where the real engine plugs in

06Step 6: run the checklist

Try it yourself

Update, not just add

Different analyser

Explain a hit

Code & diagrams

Break it on purpose

Update without removing the old version

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can