PHASE 1Beginner ~23 min· topic 1 of 9

Topic 1.1

From Text to Terms

In one line

Before a search engine can compare a question with a document it turns both into the same clean list of terms: lower-case, no punctuation, maybe without small words and with word endings trimmed. Every one of these choices helps some searches and hurts others.

Think of it like this

A library card catalogue. The librarian does not file a book under the exact way someone might say its title out loud. She writes a clean, standard heading on the card, so "Cycling", "cycling!" and "CYCLING" all land in the same drawer. Turning text into terms is writing that clean heading, for every document and for every question, in exactly the same way.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Term: One searchable unit after analysis, usually a lower-cased word.
Analysis: The pipeline that turns raw text into terms: split, lower-case, filter, stem.
Tokenise: Cut text into pieces. Here: runs of letters and digits, lower-cased.
Stop word: A very common word ("the", "of") that is often dropped because it says little.
Stemming: Chopping word endings by rules so related forms share one term (rides, riding to ride).
Normalisation (Unicode): Writing the same character in one standard form, so composed and decomposed accents compare equal.
Folding: Removing accents and case differences so café and Cafe become cafe.
Recall and precision: Recall: how much of what you wanted was found. Precision: how much of what was found you wanted.

Step by step

01Step 1: the problem, one capital letter

Compare "Refunds" with "refunds" in code and you get False. A search that compares raw strings fails on capitals, punctuation and word forms before it even starts. Analysis fixes this once, in one place.

problem.pywhole filepython

print("Refunds" == "refunds")        # False
print("cost?" == "cost")             # False

02Step 2: lower-case and split

Lower-case first, then take every run of letters and digits. Keep digits: 15 minutes and 7 am matter in a handbook.

tokenize.pywhole filepython

import re

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

print(tokenize("Shops open at 7 AM!"))
# ['shops', 'open', 'at', '7', 'am']

Step 2: lower-case and splitdiagram

Rendering diagram…

03Step 3: drop the small words

Keep a set of stop words and filter them out. In the example, are, for, every, after and the disappear from the sentence about late returns. What is left is the content: bikes returned late charged 2 dollars 15 minutes booked end time.

04Step 4: stem with simple rules

Four rules are enough to see the idea: ies to y, drop ing, drop ed, drop a plural s (but not ss). Each rule has a length guard so short words survive. Run the example and read the last four lines: charged and charges still do not meet, which is the lesson.

05Step 5: accents and special letters

The second example shows the Unicode trap. The raw tokeniser splits Café into caf and nothing, and Straße into stra and e. Folding (normalise to decomposed form, remove the accent marks, lower-case, map ß to ss) gives cafe and strasse, and the query cafe now matches.

06Step 6: two spellings of one character

caf\u00e9 (4 characters) and cafe\u0301 (5: an e plus a combining accent) look identical on screen but are not equal strings. Search engines normalise before comparing. The example prints both lengths and shows they are equal only after folding.

07Step 7: the golden rule

Analyse documents and questions with the same function. The last line of the first example shows what a stemming mismatch costs: the question Charges for late bikes shares only late and bike with the document, because the document says charged. Decide your analysis once, write it once, and call it from both sides.

Try it yourself

1
Predict, then run
Before running the first example, write down what stem("charging") and stem("cancelling") return. Add them to a group and run. Which one is wrong, and why?
2
Fix one rule
Make stem turn cancelled into cancel by also removing a doubled final letter after ed. Does it break any other word in the groups? Every rule you add can break another word, which is why real stemmers are tested on large word lists.
3
Change the stop list
Remove late from the sentence by adding it to STOP and read the stems: line. Then imagine a user searching for "late" in the handbook. What did you just make impossible?
4
Compare languages
Run the accents example in all three tabs. They print identical output although each language normalises text with a different library call (unicodedata, String.normalize, Normalizer).

Code & diagrams

Stop words and a tiny stemmer New tab

This stemmer is deliberately crude so you can read all of it. Real stemmers (Porter, Snowball) have many more rules and still make mistakes like these.

Expected output

tokens:  bikes returned late are charged 2 dollars for every 15 minutes after the booked end time
no stop: bikes returned late charged 2 dollars 15 minutes booked end time
stems:   bike return late charg 2 dollar 15 minute book end time
charged=charg, charges=charge, charge=charge
cancelled=cancell, cancellations=cancellation, cancel=cancel
rides=ride, riding=rid, ride=ride
bus=bus, news=new, series=sery
query 'Charges for late bikes' shares: late bike

Accents and case: fold before you compare New tab

Written with \u escapes so the source stays plain ASCII. In your own code you can type é directly.

Expected output

raw tokens:    caf | on | stra | e | 7
folded tokens: cafe | on | strasse | 7
lengths: 4 and 5
equal as written: no
equal after folding: yes
query 'cafe' finds the raw text: no
query 'cafe' finds the folded text: yes
a collision: ano = ano

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Stem every word that ends in s

In the Python tab of the first example, change the last rule of stem to if word.endswith("s"): (remove the length guard and the ss check). Run.

terminal

$ Run

── what you'll see ──

charged=charg, charges=charge, charge=charge

cancelled=cancell, cancellations=cancellation, cancel=cancel

rides=ride, riding=rid, ride=ride

bus=bu, news=new, series=sery

Break #2

Skip the accent step

In the Python tab of the second example, replace the line text = unicodedata.normalize("NFD", text) with text = text and run.

terminal

$ Run

── what you'll see ──

raw tokens: caf | on | stra | e | 7

folded tokens: caf | on | strasse | 7

lengths: 4 and 5

equal as written: no

equal after folding: no

query 'cafe' finds the raw text: no

query 'cafe' finds the folded text: no

Myth vs fact

Myth

Stemming makes search smarter, so always turn it on.

Fact

Stemming trades precision for recall and makes visible mistakes (news becomes new). It helps where word forms vary a lot and queries are short. It can hurt names, codes and technical terms.

Myth

Stop words are always useless, so always remove them.

Fact

They carry no topic, but they matter in phrases ("to be or not to be") and in short queries. BM25 already gives common words little weight, which is why many modern engines keep them.

Myth

Search terms and LLM tokens are the same thing.

Fact

A model's tokens are sub-word pieces for reading any text. Search terms are whole words chosen for matching. Keyword search needs no model tokenizer.

Myth

Folding accents is always an improvement.

Fact

It helps people who type without accents, but it merges words that differ in meaning in some languages. Decide per language and per product.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Analysis choices are part of the index, not of the query. Change the stemmer or the stop list and every stored term is stale: you must re-index everything. Version your analyser and store the version with the index (Topic 11.8 does the same for embedding models).
▸
Language matters more than any rule. English stemmers do nothing useful for Chinese or Japanese (no spaces between words), and German needs decompounding (Fahrradschloss is bike plus lock). Engines such as Lucene ship per-language analysers; do not reuse the English one for other text.
▸
For debugging, make the analyser explain itself: print the terms a document and a question produce. In real incidents, a large share of "search is broken" turns out to be "the query was analysed differently from the documents".

Remember this

1
Why this step exists. A computer compares strings letter by letter. To it, Refunds, refunds and refunds. are three different things, so a question typed in lower case would never find a passage that begins with a capital. Search needs a step that makes equivalent spellings identical. That step is called analysis, and the pieces it produces are terms.
2
Tokenising. The first job is to cut text into pieces. The simplest rule, used all through this course, is: lower-case everything, then keep each run of letters and digits as one term. "Shops open at 7 am!" becomes shops, open, at, 7, am. Punctuation and spaces only separate terms. This is the tokenize function from Phase 0, and every example in this phase uses it.
3
Stop words. Words such as the, is, of and at appear in almost every passage, so they say little about what a passage is about. A list of such words, the stop list, can be removed before indexing. That makes the index smaller and stops common words from adding noise. The cost: a query made only of stop words ("to be or not to be") finds nothing, and some small words matter in phrases. Modern scoring (Topic 1.3 and 1.4) already gives common words very little weight, so many engines keep them.
4
Stemming. ride, rides, riding all talk about the same thing. A stemmer chops word endings by rules so they share one term. A tiny stemmer can be written in ten lines (the example does this), and it shows the problem with all stemmers: rules are blunt. charged becomes charg but charges becomes charge, so they still do not match. riding becomes rid. news becomes new. Real stemmers (Porter, Snowball) have many more rules and still make mistakes. Stemming trades precision (finding only what you meant) for recall (finding everything related).
5
Unicode traps. Real text is not just a-z. With the simple rule above, Café is split into caf and nothing, because é is not in a-z. The fix has two parts. First, normalise: the same visible é can be stored as one character (composed) or as e plus a combining accent (decomposed), and the two are not equal as strings. Second, fold: strip the accents and map special letters (ß to ss) so café and cafe match. Folding also merges words that differ in meaning (Spanish año "year" and ano "anus" both become ano), so a multilingual product must decide per language.
6
Search terms are not an LLM's tokens. A language model also cuts text into "tokens" (LLM Engineering Topic 1.1), but those are sub-word pieces chosen so the model can read any text, and a word like unbelievable may be three of them. Search terms are whole words chosen so a query can find a document. They are different tools with different goals, and you do not need a model's tokenizer to run keyword search.
7
The golden rule. Whatever analysis you apply to documents when you build the index, you must apply to every question at search time, character for character. If the index holds charg and the query is stemmed differently, a perfectly good match is silently missed. Topic 1.7 shows this exact bug, and the playbook (Topic 1.9) makes it impossible by using one analyser function for both.

Explain it without notes

What is analysis, and why must the same analysis be applied to documents and to questions?

What does a stemmer do, and what two kinds of mistake does it make?

Why does Café break a simple [a-z0-9]+ tokeniser, and what are the two steps that fix it?

How are search terms different from an LLM's tokens?

Practice

Write analyze(text, stop) that lower-cases, splits into runs of letters and digits, removes the words in the set stop, and returns the list. Print the result for "The monthly pass costs 25 dollars" with stop = {"the"}.

Write stem_plural(word) that removes a final s only when the word is longer than 3 letters and does not end in ss. Print it for rides, bus, glass and dollars.

Trade-offs

↔
Aggressive analysis (stop words, stemming, folding): higher recall and a smaller index, at the cost of precision and sometimes surprising matches.
↔
Minimal analysis (lower-case and split only): predictable and easy to debug, but misses rides when the user typed ride.
↔
One analyser for all languages: simple to run, wrong for most non-English text. Per-language analysers are better and need language detection.

Done when you can

Done when you can write tokenize and explain every step it does.
Done when you can explain one benefit and one danger of stop words and of stemming.
Done when you can show with an example why charged and charges fail to match under a crude stemmer.
Done when you can explain why café needs normalising and folding, and when folding is harmful.
Done when you can state the golden rule: the same analysis at index time and query time.

Command Palette

Words you'll meet

Step by step

01Step 1: the problem, one capital letter

02Step 2: lower-case and split

03Step 3: drop the small words

04Step 4: stem with simple rules

05Step 5: accents and special letters

06Step 6: two spellings of one character

07Step 7: the golden rule

Try it yourself

Predict, then run

Fix one rule

Change the stop list

Compare languages

Code & diagrams

Break it on purpose

Stem every word that ends in s

Skip the accent step

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can