The 4-characters rule against a tokenizer
The vocabulary is made up for the demo. With a real tokenizer the gaps look similar: the rule is fine for plain English and weak for numbers, code and other languages.
Tokens decide your cost, your limits and your speed, so you need to count them. The 4-characters-per-token rule is a fine first guess and a bad final answer: measure with the model's own tokenizer or read the usage field of the reply.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write fits(text, limit, tokens_of) that returns True when tokens_of(text) is at most limit. Test it with tokens_of = len on a 10-character text with limits 10 and 9.
Write batch_cost(token_counts, price_per_mtok) that returns the dollar cost of all the counts together, printed with 6 decimals. Use counts [1000, 2500, 500] and price 0.50.
Explain it without notes
Why do you need to count tokens and not characters or words?
When is the 4-characters-per-token rule good, and when does it fail?
What are the three ways to know a call's token count and how exact is each?
Expected output
hello world chars=11 estimate= 3 actual= 3
internationalization chars=20 estimate= 5 actual= 3
1234567 chars= 7 estimate= 2 actual= 4
for i in range(3): chars=18 estimate= 5 actual=11
zqxjkvw chars= 7 estimate= 2 actual= 7
total actual tokens: 28
cost of one batch at the example price: 0.000014 dollars