PHASE 1Beginner ~18 min· topic 2 of 10

Topic 1.2

Counting Tokens

In one line

Tokens decide your cost, your limits and your speed, so you need to count them. The 4-characters-per-token rule is a fine first guess and a bad final answer: measure with the model's own tokenizer or read the usage field of the reply.

Think of it like this

A taxi fare. The meter charges by distance, not by the number of words you say to the driver. If you guess the fare from the number of streets, you will sometimes be far off. You want to see the meter. For models, the token count is the meter.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Input tokens: Tokens you send: system prompt, history, documents, the question. Also called prompt tokens.
Output tokens: Tokens the model writes back. Also called completion tokens. Usually the pricier kind.
Usage: The part of an API reply that reports the real token counts for that call.
Rule of thumb: A quick estimate that is usually close but can be wrong. For English, about 4 characters per token.
Price per million tokens: How providers quote prices. Divide by 1,000,000 to get the price of one token.

Step by step

01Step 1: the guess

The simplest estimate is len(text) / 4. It costs nothing and is good enough to say "this paragraph is about 100 tokens".

guess.pywhole filepython

def estimate(text):
    return (len(text) + 3) // 4

print(estimate("hello world"))

02Step 2: build something to compare against

To see where the guess fails, the example uses a tiny made-up tokenizer: a short vocabulary and a greedy rule (take the longest piece that matches here, otherwise one character). It is not a real tokenizer, but it behaves like one: common pieces are short ids, strange text is split into single characters.

03Step 3: compare on five kinds of text

Run it. Plain words match the estimate. internationalization is overestimated (5 against 3) because its pieces are all in the vocabulary. 1234567 and zqxjkvw are underestimated: digit groups and gibberish split into many pieces. The estimate can be wrong in both directions.

04Step 4: turn tokens into money

Cost is tokens * price_per_million / 1,000,000. Keep the price as a named variable (PRICE_IN_PER_MTOK), because prices change and differ per model. The example prices one batch of the five texts at an example price.

Step 4: turn tokens into moneydiagram

Rendering diagram…

05Step 5: trust the usage field after the call

Before the call, count with the right tokenizer to decide whether to send. After the call, read usage.prompt_tokens and usage.completion_tokens, log them, and compare with your own count. If they differ a lot, the wrong tokenizer or hidden extra tokens (the chat format adds a few per message) are the cause.

Try it yourself

1
Find where the rule fails
Add "def add(a, b): return a + b" and a number such as "9876543210" to texts. Predict the estimate and the actual count first. Which direction is the error?
2
Change the price
Set the example price to 2.00 and the batch to 100,000 copies of the five texts (multiply total by 100,000). What is the cost? Why would you cache or trim a prompt that is sent that often?
3
Extend the vocabulary
Add "zq" and "xj" to the vocabulary and re-run. Only the gibberish line changes: tokenizers are tuned to the text they were trained on.

Code & diagrams

The 4-characters rule against a tokenizer New tab

The vocabulary is made up for the demo. With a real tokenizer the gaps look similar: the rule is fine for plain English and weak for numbers, code and other languages.

Expected output

hello world            chars=11 estimate= 3 actual= 3
internationalization   chars=20 estimate= 5 actual= 3
1234567                chars= 7 estimate= 2 actual= 4
for i in range(3):     chars=18 estimate= 5 actual=11
zqxjkvw                chars= 7 estimate= 2 actual= 7
total actual tokens: 28
cost of one batch at the example price: 0.000014 dollars

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Count characters instead of tokens for a limit

In the Python version, replace actual = len(tokenize(text)) with actual = estimate(text) and run.

terminal

$ Run

── what you'll see ──

hello world chars=11 estimate= 3 actual= 3

internationalization chars=20 estimate= 5 actual= 5

1234567 chars= 7 estimate= 2 actual= 2

for i in range(3): chars=18 estimate= 5 actual= 5

zqxjkvw chars= 7 estimate= 2 actual= 2

total actual tokens: 17

cost of one batch at the example price: 0.000008 dollars

Myth vs fact

Myth

One token is one word.

Fact

On average a word is a bit more than one token in English, and much more in other languages or for code and numbers.

Myth

I can count tokens with any tokenizer.

Fact

Counts differ between model families. Use the tokenizer that matches the model you call, or read the usage field.

Myth

Only the question counts.

Fact

The system prompt, the whole history, any retrieved documents and the reply all count.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Chat formats add a few tokens of overhead per message (role markers and separators). Provider guides give the exact formula per model, and the usage field includes them, so an exact count of your text alone is a few tokens short.
▸
Log prompt_tokens and completion_tokens per request and per feature. Cost spikes almost always trace to one prompt getting longer, which only per-feature logs reveal (Topic 11.4).
▸
Images, audio and tool definitions are also converted to tokens. A long list of tool schemas can cost hundreds of input tokens on every call, even when no tool is used.

Remember this

1
Why count at all. Three things are measured in tokens: cost (providers charge per million input and output tokens), limits (the context window and the per-minute token rate limit) and speed (the reply is produced one token at a time). Without counts you cannot predict a bill, cannot tell whether a prompt fits, and cannot size a rate limit.
2
The rule of thumb. For ordinary English prose, one token is about four characters, or three quarters of a word. It is useful to estimate a budget in a meeting. It is wrong for numbers (digit groups are short), code (symbols and indentation), non-English text, and long rare words. The example prints a made-up tokenizer next to the estimate: for 1234567 and for random letters the estimate is too low, for internationalization it is too high.
3
Each model has its own tokenizer. A tokenizer from one family does not count tokens for another. Even different versions of the same provider's models can differ. Using the wrong one can be off by several percent or more, and the error is largest for code, numbers and non-English text. Keep a safety margin when you count with anything other than the model's own tokenizer.
4
Three ways to measure, from cheap to exact. (1) The rule of thumb, for sizing ideas. (2) The provider's tokenizer library or a token-counting endpoint, to count before sending. (3) The usage object in every API reply (prompt_tokens, completion_tokens), which is the number you are billed on, so log it on every call. The first is a guess, the third is the truth.
5
Input and output are priced and limited separately. Output tokens usually cost more than input tokens and are produced slowly, so a long reply is both the most expensive and the slowest part. Count both, and set max_tokens (the cap you choose on how long the reply may be) so a runaway reply cannot blow the budget (Topic 6.1). The context window (how many tokens fit in one call) and rate limits (how many tokens you may send per minute) are explained in Topics 1.5 and 6.7.

Explain it without notes

Why do you need to count tokens and not characters or words?

When is the 4-characters-per-token rule good, and when does it fail?

What are the three ways to know a call's token count and how exact is each?

Practice

Write fits(text, limit, tokens_of) that returns True when tokens_of(text) is at most limit. Test it with tokens_of = len on a 10-character text with limits 10 and 9.

Write batch_cost(token_counts, price_per_mtok) that returns the dollar cost of all the counts together, printed with 6 decimals. Use counts [1000, 2500, 500] and price 0.50.

Trade-offs

↔
Counting before every call costs a little CPU, and saves failed or oversized requests.
↔
A safety margin wastes a small share of the window, and protects against counting error.
↔
Local tokenizer libraries need updating when models change, while the usage field is always right but arrives late.

Done when you can

Done when you can convert tokens to dollars for input and output separately.
Done when you can name three cases where the 4-characters rule fails.
Done when you log prompt and completion tokens on every call.
Done when you can explain why the tokenizer must match the model.

Command Palette

Words you'll meet

Step by step

01Step 1: the guess

02Step 2: build something to compare against

03Step 3: compare on five kinds of text

04Step 4: turn tokens into money

05Step 5: trust the usage field after the call

Try it yourself

Find where the rule fails

Change the price

Extend the vocabulary

Code & diagrams

Break it on purpose

Count characters instead of tokens for a limit

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can