Tokens, Context Windows, Cost and Rate Limits

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Token

A small piece of text the model reads or writes. Roughly four English characters, but it varies.

Context window

The most tokens a model can handle in one call, counting input and output together.

Input / output tokens

Tokens you send in, and tokens the model generates. They are priced separately.

Rate limit

A cap on how many requests or tokens you may use per minute.

HTTP 429

The status code for "too many requests, slow down".

Exponential backoff

Waiting longer after each failure: 0.5s, 1s, 2s, 4s and so on.

Jitter

A random amount added to a wait so many clients do not retry at the same moment.

Fallback

A second choice (another model) used when the first fails.

1. 1

   Shrink the budget

   In the trim example set budget to 25. Predict which messages remain, then run. Is the system message ever dropped?

2. 2

   Make a model fail more

   In the backoff example set failures=9 for the primary model and retries=2. Predict which lines print, then check.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

LIMIT = 20


def call_model(prompt_tokens):
    if prompt_tokens > LIMIT:
        raise ValueError(f"context window exceeded: {prompt_tokens} tokens > limit {LIMIT}")
    return "ok"


print(call_model(15))
print(call_model(52))

1. 1

   A token is the small piece of text a model actually reads: often a whole common word, part of a rare word, a punctuation mark or a space plus a word. As a rough rule of thumb in English, a token is about four characters, or three quarters of a word. This is only an approximation, and it is different for code, other languages and every model family.

2. 2

   A tokenizer is the program that cuts text into tokens. Real tokenizers (commonly byte-pair encoding, which learns the most common pieces from lots of text) are trained, not hand-written, and each model has its own. The toy tokenizer in this topic cuts words into four-letter pieces to show the idea. Never use a toy count for real billing or limits. Use the provider's tokenizer library or its token-counting endpoint, and rely on the usage numbers returned in each response.

3. 3

   The context window is the maximum number of tokens the model can handle in one call. Your input (system text, history, retrieved documents: extra text you look up and paste in to help the model, Topic 16.9) and the model's output share it. If the input is too long the API returns an error. If the conversation grows each turn, sooner or later it does not fit, so you must decide what to drop or summarise. Window sizes differ between models and change over time: always read the current documentation for the model you use.

4. 4

   Cost is a formula. Providers typically charge separately per million input tokens and per million output tokens, with output usually priced higher. So: cost = input_tokens / 1,000,000 × input_rate + output_tokens / 1,000,000 × output_rate. The rates in this topic are made-up examples only. Read your provider's current price page, and remember a chat resends its history. Turn 1 sends one message, turn 2 sends two, turn 10 sends ten. So a long conversation's total input tokens grow roughly with the square of its length.

5. 5

   Rate limits cap how many requests or tokens you may send per minute. When you exceed them the API answers with HTTP status 429 (Too Many Requests), often with a Retry-After header (an extra labelled line of information that comes with the response; Topic 15.1). The right reaction is to wait and try again, with exponential backoff (wait 0.5s, 1s, 2s, 4s...) plus jitter (a random part so many clients do not all retry at the same instant). You met this in Topic 15.2; here it guards model calls.

6. 6

   A fallback list adds resilience: if the preferred model is overloaded or failing, try the next model in a list, perhaps a smaller or cheaper one. Combine a retry limit, backoff, a fallback list and a budget guard that refuses to start a call that clearly will not fit. The Hectal tutor uses exactly these pieces: streaming answers, daily limits and model fallbacks.

• ↔

  Trim hard or keep context. Aggressive trimming is cheap and fast but can lose facts. Keeping everything is accurate but costs more and hits the window. Summaries sit in between and cost one extra call.

• ↔

  Retry more or fail fast. More retries survive short outages but make users wait. Chat interfaces usually want one or two retries and then a fallback or a clear message.

• Done when you can explain tokens, context window and why they are not characters or words.

• Done when you can write the cost formula and compute an example with given rates.

• Done when you can trim a message list to a budget while keeping the system message.

• Done when you can write retry-with-backoff and say which errors to retry.

• Done when you know a full Hectal course on LLM engineering is planned to cover production budgets, caching and routing.