Stop words and a tiny stemmer
This stemmer is deliberately crude so you can read all of it. Real stemmers (Porter, Snowball) have many more rules and still make mistakes like these.
Before a search engine can compare a question with a document it turns both into the same clean list of terms: lower-case, no punctuation, maybe without small words and with word endings trimmed. Every one of these choices helps some searches and hurts others.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write analyze(text, stop) that lower-cases, splits into runs of letters and digits, removes the words in the set stop, and returns the list. Print the result for "The monthly pass costs 25 dollars" with stop = {"the"}.
Write stem_plural(word) that removes a final s only when the word is longer than 3 letters and does not end in ss. Print it for rides, bus, glass and dollars.
Explain it without notes
What is analysis, and why must the same analysis be applied to documents and to questions?
What does a stemmer do, and what two kinds of mistake does it make?
Why does Café break a simple [a-z0-9]+ tokeniser, and what are the two steps that fix it?
How are search terms different from an LLM's tokens?
Expected output
tokens: bikes returned late are charged 2 dollars for every 15 minutes after the booked end time
no stop: bikes returned late charged 2 dollars 15 minutes booked end time
stems: bike return late charg 2 dollar 15 minute book end time
charged=charg, charges=charge, charge=charge
cancelled=cancell, cancellations=cancellation, cancel=cancel
rides=ride, riding=rid, ride=ride
bus=bus, news=new, series=sery
query 'Charges for late bikes' shares: late bike