What the model sees for strawberry
The ids are made up for the demo. Real ids depend on the model's own vocabulary.
A token is the small piece of text a model reads and writes: often a whole common word, part of a rare word, a digit group or a symbol. Models work on token ids, not letters, which explains the bill, the limits and why spelling questions are hard.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write pair_counts(symbols) that returns a dictionary counting every pair of neighbouring symbols, keyed as "a b". Run it on list("lower").
Write greedy_tokens(text, vocab) that cuts text into the longest matching vocabulary pieces from the left, falling back to one character. Run it on "strawberry" with vocab ["str", "straw", "berry", "aw"] and print the pieces joined with | .
Explain it without notes
Why do language models use tokens instead of characters or whole words?
Explain byte-pair encoding in your own words.
Why is a model unreliable at counting letters in a word, and what is the engineering fix?
Why does the same sentence cost different amounts in different languages and models?
Expected output
what you see: str | aw | berry
what the model gets: 3504 675 19772
id 3504 is one unit, 3 letters inside, no letters visible
id 675 is one unit, 2 letters inside, no letters visible
id 19772 is one unit, 5 letters inside, no letters visible
letter r counted by code: 3
letter r counted by the model: it must have learned which ids contain an r