Accents and case: fold before you compare
Written with \u escapes so the source stays plain ASCII. In your own code you can type é directly.
Before a search engine can compare a question with a document it turns both into the same clean list of terms: lower-case, no punctuation, maybe without small words and with word endings trimmed. Every one of these choices helps some searches and hurts others.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write analyze(text, stop) that lower-cases, splits into runs of letters and digits, removes the words in the set stop, and returns the list. Print the result for "The monthly pass costs 25 dollars" with stop = {"the"}.
Write stem_plural(word) that removes a final s only when the word is longer than 3 letters and does not end in ss. Print it for rides, bus, glass and dollars.
Explain it without notes
What is analysis, and why must the same analysis be applied to documents and to questions?
What does a stemmer do, and what two kinds of mistake does it make?
Why does Café break a simple [a-z0-9]+ tokeniser, and what are the two steps that fix it?
How are search terms different from an LLM's tokens?
Expected output
raw tokens: caf | on | stra | e | 7
folded tokens: cafe | on | strasse | 7
lengths: 4 and 5
equal as written: no
equal after folding: yes
query 'cafe' finds the raw text: no
query 'cafe' finds the folded text: yes
a collision: ano = ano