PHASE 1Beginner ~22 min· topic 1 of 10

Topic 1.1

Tokens

In one line

A token is the small piece of text a model reads and writes: often a whole common word, part of a rare word, a digit group or a symbol. Models work on token ids, not letters, which explains the bill, the limits and why spelling questions are hard.

Think of it like this

LEGO bricks. You can build a house from single 1-by-1 bricks, but it takes forever. So the box also contains big wall pieces and window pieces for the shapes people build most often. A tokenizer is that box: it cuts text into the biggest ready-made pieces it knows, and falls back to small ones for unusual words.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Token: One piece of text the model reads or writes: a word, part of a word, some digits or a symbol.
Tokenizer: The program (and its piece list) that cuts text into tokens and back.
Vocabulary: The full list of pieces a model knows. Each piece has a number, its token id.
Token id: The number that stands for a piece. The model sees ids, never letters.
BPE: Byte-pair encoding: build the vocabulary by repeatedly merging the most common neighbouring pair.
Merge: One step of BPE: two neighbouring pieces become one new piece.
Corpus: A pile of text used to learn from. Here it is four words with counts. For a real tokenizer it is a large part of the internet.

Step by step

01Step 1: letters are too small, words are too big

Try spelling a sentence with one id per letter: a 100-word paragraph becomes about 600 ids. Try one id per word: you need a dictionary with every word, name and typo ever written. Neither is good, so we want pieces in between, chosen from data.

02Step 2: count neighbouring pairs

Take a small corpus (a pile of text to learn from): low (5 times), lower (2), newest (6), widest (3). Write each word as separate letters. Count every pair of neighbours, weighted by how often the word appears. The pair e s appears in newest (6) and widest (3), so it scores 9. In the real thing the corpus is a large part of the internet.

count.pywhole filepython

words = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
counts = {}
for w, n in words.items():
    for a, b in zip(w, w[1:]):
        counts[a + " " + b] = counts.get(a + " " + b, 0) + n
print(sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:3])

03Step 3: merge the winner and repeat

Glue the best pair into one piece everywhere it appears, then count again. The first example prints six merges: the order depends only on counts (ties go to the pair that sorts first). After six merges, frequent words such as newest are one or two pieces while widest is still partly letters. That is exactly how rare words end up as several tokens.

Step 3: merge the winner and repeatdiagram

Rendering diagram…

04Step 4: encode new text with the learned merges

To tokenize new text, a real tokenizer applies the learned merges in the order they were learned (or picks the longest known piece). Anything unknown falls back to smaller pieces, down to single bytes, so no text is ever impossible to encode. That is why modern models never print an "unknown word" error.

05Step 5: see what the model actually receives

The second example splits strawberry into three pieces and shows the ids the model would get (the ids here are made up). Counting r with code is trivial. The model has to remember which ids contain an r, which is why letter questions are error-prone and why you should use code for them.

06Step 6: what to do with this as an engineer

Never assume a token is a word or a character. Budget in tokens, count with the right tokenizer (Topic 1.2), put exact work such as counting, sorting letters and arithmetic into code, and expect non-English text and code to cost more tokens than the same English sentence.

Try it yourself

1
Predict the first merge
Before running the first example, count the pairs by hand from the corpus and write down which pair wins the first merge and how many times it was seen. Then run it and compare.
2
Change the corpus
Change widest to appear 10 times instead of 3. Which merges change? Frequent words get their own pieces first, so what happens to widest now?
3
More merges
Change range(1, 7) (or the loop count in JavaScript and Java) to 12 merges. How many pieces does each word end with? Is any word a single piece?

Code & diagrams

Byte-pair encoding on a tiny corpus New tab

Real tokenizers do the same thing with bytes, a huge corpus and tens of thousands of merges. Each language uses its own idiom (dict, Map, LinkedHashMap) for the same steps.

Expected output

merge 1: e + s -> es (seen 9 times)
merge 2: es + t -> est (seen 9 times)
merge 3: l + o -> lo (seen 7 times)
merge 4: lo + w -> low (seen 7 times)
merge 5: e + w -> ew (seen 6 times)
merge 6: ew + est -> ewest (seen 6 times)
low    = low
lower  = low | e | r
newest = n | ewest
widest = w | i | d | est

What the model sees for strawberry New tab

The ids are made up for the demo. Real ids depend on the model's own vocabulary.

Expected output

what you see:          str | aw | berry
what the model gets:   3504 675 19772
id  3504 is one unit, 3 letters inside, no letters visible
id   675 is one unit, 2 letters inside, no letters visible
id 19772 is one unit, 5 letters inside, no letters visible
letter r counted by code: 3
letter r counted by the model: it must have learned which ids contain an r

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Break the tie rule

In the Python version of best_pair, flip the tie-break so that ties go to the pair that sorts LAST: replace the min(counts, key=lambda k: (-counts[k], k)) line with max(sorted(counts, reverse=True), key=lambda k: counts[k]).

terminal

$ Run

── what you'll see ──

merge 1: s + t -> st (seen 9 times)

merge 2: e + st -> est (seen 9 times)

merge 3: o + w -> ow (seen 7 times)

merge 4: l + ow -> low (seen 7 times)

merge 5: w + est -> west (seen 6 times)

merge 6: n + e -> ne (seen 6 times)

low = low

lower = low | e | r

newest = ne | west

widest = w | i | d | est

Myth vs fact

Myth

A token is a word.

Fact

A token is a piece. Common words are one token, rare words are several, and numbers, punctuation and spaces count too.

Myth

The model reads letters, so it should be able to count them.

Fact

It reads ids of pieces. Letter counting works only when it has learned the spelling of that piece, so use code for it.

Myth

All models count tokens the same way.

Fact

Each model family has its own vocabulary, so the same text gives different counts in different models.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Production tokenizers usually work on bytes, not characters, so any input (emoji, rare scripts) can be encoded without an unknown token. A single character can therefore be several tokens.
▸
The vocabulary size is a trade-off: a bigger vocabulary means fewer tokens per text (cheaper, faster) but larger embedding tables and rarer pieces that are poorly learned.
▸
Whitespace and capitalisation change tokenization: Hello, Hello and hello can be three different tokens. Prompt templates that differ by one space can change behaviour slightly.

Remember this

1
Why tokens exist. A model is a function over numbers, so text must become numbers first. Using one number per letter makes every sentence very long, and long inputs are slow and expensive. Using one number per word needs a giant dictionary and fails on new words, names and typos. Pieces in between solve both: common words are one piece, rare words are a few pieces, and anything can be spelled out from small pieces.
2
Byte-pair encoding (BPE). The most common way to build the piece list. Start with single characters. Count which two neighbouring pieces appear together most often in a big pile of text, glue them into one new piece, and repeat thousands of times. The first merges make pieces like er and ing, later merges make whole words. The example below does six merges on four words by hand. Each piece gets an id (its position in the list), and the model only ever sees ids.
3
What tokens look like in practice. Common English words are usually one token, often with the leading space included. Long or rare words split into several pieces. Numbers split into groups of digits, so 1234567 is several tokens. Code is token-hungry because of symbols and indentation. Text in languages that were rare in the training data usually needs more tokens for the same meaning, which makes it slower and costlier. Every model family has its own piece list, so the same text has different token counts in different models (Topic 1.2).
4
Why models are bad at spelling and counting letters. The model receives ids, not letters. The id for the piece berry carries no list of its letters. Asking "how many r in strawberry" is like handing someone three sealed boxes labelled str, aw and berry and asking how many r letters are inside, without letting them open the boxes. They can answer only if they happen to remember what is in each box. A model can learn the answer for common words, but it is unreliable. Code counts letters perfectly (Topic 0.3).
5
Tokens are the unit of everything you pay for and are limited by. Price is per token, the context window is a number of tokens (Topic 1.5), speed is tokens per second, and rate limits count tokens per minute. When someone says "the model has 8,000 of context" they mean tokens. Keep this rule of thumb in mind: 1 token is roughly 3 to 4 characters of English, or about three quarters of a word, but measure instead of guessing (Topic 1.2).

Explain it without notes

Why do language models use tokens instead of characters or whole words?

Explain byte-pair encoding in your own words.

Why is a model unreliable at counting letters in a word, and what is the engineering fix?

Why does the same sentence cost different amounts in different languages and models?

Practice

Write pair_counts(symbols) that returns a dictionary counting every pair of neighbouring symbols, keyed as "a b". Run it on list("lower").

Write greedy_tokens(text, vocab) that cuts text into the longest matching vocabulary pieces from the left, falling back to one character. Run it on "strawberry" with vocab ["str", "straw", "berry", "aw"] and print the pieces joined with | .

Trade-offs

↔
A large vocabulary shortens text but makes the model bigger and leaves rare pieces under-trained.
↔
Byte-level pieces never fail on strange input but can split unusual characters into many tokens.
↔
Counting tokens with the real tokenizer is exact but adds a dependency, while a rule of thumb is free and wrong in the cases that cost most.

Done when you can

Done when you can explain why models use sub-word tokens.
Done when you can run byte-pair encoding by hand on a tiny corpus.
Done when you can explain why letter counting is unreliable and what to use instead.
Done when you can say which kinds of text cost more tokens.

Command Palette

Words you'll meet

Step by step

01Step 1: letters are too small, words are too big

02Step 2: count neighbouring pairs

03Step 3: merge the winner and repeat

04Step 4: encode new text with the learned merges

05Step 5: see what the model actually receives

06Step 6: what to do with this as an engineer

Try it yourself

Predict the first merge

Change the corpus

More merges

Code & diagrams

Break it on purpose

Break the tie rule

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can