Byte-pair encoding on a tiny corpus
Real tokenizers do the same thing with bytes, a huge corpus and tens of thousands of merges. Each language uses its own idiom (dict, Map, LinkedHashMap) for the same steps.
A token is the small piece of text a model reads and writes: often a whole common word, part of a rare word, a digit group or a symbol. Models work on token ids, not letters, which explains the bill, the limits and why spelling questions are hard.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write pair_counts(symbols) that returns a dictionary counting every pair of neighbouring symbols, keyed as "a b". Run it on list("lower").
Write greedy_tokens(text, vocab) that cuts text into the longest matching vocabulary pieces from the left, falling back to one character. Run it on "strawberry" with vocab ["str", "straw", "berry", "aw"] and print the pieces joined with | .
Explain it without notes
Why do language models use tokens instead of characters or whole words?
Explain byte-pair encoding in your own words.
Why is a model unreliable at counting letters in a word, and what is the engineering fix?
Why does the same sentence cost different amounts in different languages and models?
Expected output
merge 1: e + s -> es (seen 9 times)
merge 2: es + t -> est (seen 9 times)
merge 3: l + o -> lo (seen 7 times)
merge 4: lo + w -> low (seen 7 times)
merge 5: e + w -> ew (seen 6 times)
merge 6: ew + est -> ewest (seen 6 times)
low = low
lower = low | e | r
newest = n | ewest
widest = w | i | d | est