Topic 6.1
Token Maths and Cost
In one line
A model call is billed by tokens: input tokens times the input price, plus output tokens times the output price. A few lines of arithmetic tell you the cost per request, per user and per month, and show why long chat histories get expensive fast.
Think of it like this
A translation bureau that charges per word you hand in to read and a higher price per word they write back. You can estimate the fee before ordering if you know roughly how long the input and the reply will be. A model call is billed the same way: one price for what you send in, another for what comes out, and a bill that adds up quietly over thousands of orders.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Token
- A small piece of text (about four English characters) that models read and write. Billing counts tokens.
- MTok
- One million tokens. Prices are written as dollars per MTok.
- Input tokens
- Everything you send in a request: system prompt, history, documents and the new question.
- Output tokens
- Everything the model writes back, including hidden reasoning tokens.
- Context window
- The most tokens a model can handle in one request, input and output together.
- usage
- The part of every reply that reports how many input and output tokens that call used.
- Micro-dollar
- A millionth of a dollar. Handy for exact whole-number cost maths with no rounding errors.
Step by step
01Step 1: the formula in four lines
Two prices and two token counts. Everything in this phase builds on this sum, so write it as one small function and keep prices as named variables, never as numbers buried in code.
PRICE_IN_PER_MTOK = 0.50 # example number
PRICE_OUT_PER_MTOK = 1.50 # example number
def cost(tokens_in, tokens_out):
return (tokens_in * PRICE_IN_PER_MTOK + tokens_out * PRICE_OUT_PER_MTOK) / 1_000_00002Step 2: one request, one user, one month
A request with 1,200 input tokens and 300 output tokens costs 1,200 × 0.50 + 300 × 1.50 = 1,050 divided by a million, which is 0.00105 dollars. Twenty requests a day for thirty days is 0.63 dollars for one user, and 10,000 users is 6,300 dollars a month. A tenth of a cent per request is not small at that volume.
03Step 3: find where the tokens go
In that request, input is 1,200 tokens × 0.50 = 600 price units, while output is 300 tokens × 1.50 = 450 price units (the same units as the 1,050 above, before dividing by a million). Even though output is the smaller number, it is 43 percent of the bill because it has a higher price. Print the split for your own feature: it tells you whether to trim the prompt or cap the answer.
04Step 4: chats re-send everything
Model a 10-turn chat: each turn sends the system prompt, the earlier exchanges and the new message, and receives one reply. The input total comes from a loop, not from a guess. Run the example and compare "send all history" with "keep last 3 turns".
05Step 5: trimming has a price too
Keeping only the last three exchanges cuts input a lot, but the model forgets what was said earlier. Topic 3.7 covers summarising old turns so you keep the meaning and drop the tokens. Always test that quality survives the trimming.
06Step 6: read usage, not estimates
Estimates like "four characters per token" are fine for planning. For bills, read usage.prompt_tokens and usage.completion_tokens from each reply and add them up in your meter (the Topic 0.7 recipe). The next topic turns that meter into guards.
Try it yourself
- 1
Predict the one-request cost
Before running, work out cost(1200, 300) with the example prices on paper (it is 0.00105). Then change the output tokens to 900 and predict again. Which part of the bill grew?
- 2
Try a cheaper history
Change
keep_lastto 1 and then to 0. Predict the input tokens for the 10-turn chat in each case, then run. At what keep value does trimming stop saving much? - 3
Price your own feature
Pick a feature you might build (a support bot, a summariser). Guess its input and output tokens, its requests per day and its users, and compute a month. Is the number a surprise?
Code & diagrams
Prices are example numbers. The chat part counts tokens turn by turn: the system prompt is re-sent every time.
Expected output
one request: 0.001050 dollars
one user, 20 requests a day for 30 days: 0.63 dollars
10,000 such users: 6300.00 dollars a month
10-turn chat, send all history: input 14050 tokens, output 1500 tokens, cost 0.009275 dollars
10-turn chat, keep last 3 turns: input 9640 tokens, output 1500 tokens, cost 0.007070 dollarsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Divide by the wrong power of ten
In cost, change the divisor from 1_000_000 to 1_000 (in Java 1_000, in JavaScript 1_000). Run it.
Myth vs fact
Myth
Cost depends on how long the answer is.
Fact
It depends on input and output together. A short answer to a very long prompt can cost more than a long answer to a short one, and hidden reasoning tokens count as output.
Myth
A chat costs the same on every turn.
Fact
Each turn re-sends the history, so later turns cost more than earlier ones unless you trim or summarise.
Myth
Tokens are the same as words.
Fact
A token is a piece of text, often part of a word. Rough rules (four characters, three quarters of a word) are for planning. The provider's usage is the truth.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Budget from the worst case, not the average: p95 prompts are often several times the median because a few users paste long documents. Cap input length at the edge (Topic 6.2).
- ▸
Per-feature cost, not just per-account: tag every call with a feature name and record
usageso you can say which feature is 70 percent of the bill (Topic 11.4). - ▸
Prompt length is a design choice, not a given. A 2,000-token system prompt sent to every request is 2,000 tokens of input on every call. At scale, trimming it or caching it (Topic 6.6) pays more than most model switches.
- ▸
Prices change and models are replaced, so a cost model is a spreadsheet you refresh, not a constant you remember.
Remember this
- 1
Why this exists. Calling a model feels free while you test (a few calls, a few cents). In a product the same call runs thousands or millions of times, and the cost grows with usage. If you can't work out the cost of one request on paper, you can't price a feature, pick a model, or notice when something is wrong. Cost is an engineering input, like latency.
- 2
The formula. A token is a small piece of text the model reads or writes (about four English characters; Topic 1.2). Providers publish two prices in dollars per million tokens (written MTok): one for input tokens (everything you send: system prompt, history, documents, the question) and one for output tokens (everything the model writes). Cost = input tokens × input price + output tokens × output price, divided by 1,000,000. Output tokens usually cost more per token than input tokens, so a long answer is expensive.
- 3
Hidden tokens count. Reasoning models write hidden thinking tokens before the answer (Topic 1.8). You don't see them in the text, but providers bill them as output tokens, and the
usageblock of the reply reports them. Always read cost fromusage, not from the length of the visible text. - 4
From request to month. Multiply up: cost per request × requests per user per day × days × users. The first example works this through with example prices. The useful habit is to do this sum before building the feature, with pessimistic numbers, and again after launch with real
usagedata. - 5
Why chats cost more than they look. The model remembers nothing between calls, so a chat app re-sends the whole conversation every turn (Topic 0.1). Turn 1 sends the system prompt and one message. Turn 10 sends the system prompt and nine earlier exchanges and the new message. The input grows with each turn, so total input over a conversation grows roughly with the square of the number of turns. The second part of the example shows keeping only the last three exchanges: the input shrinks a lot, and the cost drops about a quarter even in a short chat.
- 6
Levers that cut cost. Send fewer input tokens (trim history, retrieve only relevant passages instead of whole documents), cap output (
max_tokens, "answer in two sentences"), use a smaller model where quality allows (Phase 13), skip the call entirely with a cache (Topics 6.4 to 6.6), and move non-urgent work to a batch API (Topic 6.10). Each lever has a quality cost, so measure before and after (Phase 9).
Explain it without notes
Write the cost formula for one model call and say which part people forget.
A chat bot costs twice what the team predicted. Name three likely causes.
Why does the total input of an untrimmed chat grow faster than the number of turns?
Why are prices kept as named variables in code?
Practice
A summariser reads 3,000 input tokens and writes 200 output tokens. With example prices of 0.40 dollars per MTok input and 1.20 per MTok output, print the cost of one call and of 50,000 calls, each with 6 decimals.
Write chat_input_tokens(turns, system, per_exchange, keep_last) returning the total input tokens of a chat. A new user message adds 50 tokens to its turn. Print it for 8 turns with a 300-token system prompt, 200 tokens per exchange, once with keep_last=None and once with keep_last=2.
Trade-offs
- ↔
Trimming history saves money but the model forgets; summaries keep meaning at the cost of an extra call.
- ↔
A cheaper model lowers the bill but may need more retries or review; compare cost per correct answer, not per call.
- ↔
Precise cost tracking needs
usageplumbing in every call; estimates are quick but drift.
Done when you can
Done when you can compute the cost of a request from token counts and two prices.
Done when you can extend it to cost per user per month.
Done when you can explain why a chat's input grows with each turn.
Done when you can name four levers that cut cost.
Done when prices in your code are named variables with example or configured values.