Topic 1.5
The Context Window
In one line
The context window is the fixed number of tokens a model can handle in one call, input and output together. The API remembers nothing between calls, so your app re-sends the history each time and must trim it when it grows too long.
Think of it like this
A desk. You can spread out only so many papers on it. To add a new paper when the desk is full, you must move an old one away. The model can only use what is on the desk right now, and when you start a new call the desk is empty again.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Context window
- The maximum number of tokens, input and output together, that a model can handle in one call.
- Stateless
- Remembering nothing between calls. Every call stands alone.
- History
- The earlier messages that your app re-sends so the model can continue a conversation.
- Truncate
- Cut text off, usually the oldest part, so the rest fits.
- Lost in the middle
- The tendency of models to use information at the start and end of a long input better than in the middle.
- Output reserve
- The part of the window you keep free for the reply.
Step by step
01Step 1: a chat is a list you re-send
Your app keeps a list of messages: system, user, assistant, user... For every turn it sends the whole list and appends the reply. The model sees the list as one text. If your app forgets to send the list, the model has no idea what "it" refers to (Topic 2.2 builds this list in code).
02Step 2: the list grows
Each turn adds a question and an answer. After many turns the list approaches the window. The example uses six messages totalling 47 "tokens" (words stand in for tokens to keep the numbers readable).
03Step 3: reserve room for the reply
With a window of 40 and 10 tokens kept for the answer, the input budget is 30. Never let the input use the whole window, or the model has no space to reply and the answer is cut short.
04Step 4: drop the oldest, keep the system and the newest
Starting after the system message, drop old messages one at a time until the total fits. Window 60 fits everything. Window 40 drops messages 1, 2 and 3. Window 20 cannot fit even the system message and the newest question (13 tokens against a budget of 10), so the right answer is an error or a shorter system prompt, not a silent cut.
# order of operations
# 1. count the tokens of every message
# 2. budget = window - reserve_for_output
# 3. drop the oldest non-system messages until the total fits
# 4. if it still does not fit, fail clearly05Step 5: choose a better trim than blind dropping
The kept messages 0, 4 and 5 contain an answer without the question that caused it. Better options: drop question and answer together, or replace the dropped part by a one-line summary written by the model. Summaries cost one extra call but keep the thread.
Try it yourself
- 1
Predict the middle case
Before running, work out for window 50 and a reserve of 10 which messages are dropped. Then change the window list to
(50,)and check. - 2
Bigger reply reserve
Change the reserve from 10 to 30 and run the three windows. Which now fail? A long answer costs window space too.
- 3
Drop in pairs
Change the trimmer in Python so it drops message 1 and 2 together, then 3 and 4. What is kept at window 40? Is the result a more sensible conversation?
Code & diagrams
Words stand in for tokens. A real version counts with the model's tokenizer.
Expected output
window 60: kept [0, 1, 2, 3, 4, 5] using 47 of 50 input tokens, dropped []
window 40: kept [0, 4, 5] using 24 of 30 input tokens, dropped [1, 2, 3]
window 20: cannot fit, needs 13 tokens but input budget is 10Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the reserve
In the Python version change the call to fit(messages, window, 0) (reserve zero tokens for the answer).
Myth vs fact
Myth
The model remembers our earlier chat.
Fact
Only if your app re-sends it. Between calls the model remembers nothing.
Myth
A bigger context window is always better.
Fact
It costs more, is slower, and long inputs bury important facts. Send the relevant text, not everything that fits.
Myth
The window only limits what I send.
Fact
It limits input plus output. A full window leaves no room to answer.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Long chats multiply cost: if every turn re-sends all history, total input tokens grow roughly with the square of the number of turns. Summarising or retrieval keeps this linear. Provider-side prompt caching can discount a repeated prefix, which favours keeping the start of the prompt stable (Phase 6).
- ▸
Keep stable content (system prompt, tool definitions, examples) at the start and variable content at the end. That helps caching and keeps instructions where models attend well.
- ▸
When you must send a long document, test whether the answer changes with the position of the key fact. If it does, retrieve the relevant passages instead of pasting the whole thing.
Remember this
- 1
Why this exists. Attention work and memory grow with the number of tokens (Topic 1.4), so each model is built and trained for a maximum length: its context window. Everything the model can use for one reply must fit: the system prompt (the standing instructions you send first), the history, retrieved documents, the question, and the reply it is about to write. Input plus output must be less than or equal to the window. For reasoning models the hidden thinking tokens count as output too (Topic 1.8).
- 2
The API is stateless. A model call is a function: tokens in, tokens out. Nothing is stored between calls. A chat "remembers" because your app sends the earlier messages again each time (Topic 2.2). So a conversation costs more with every turn: turn 10 re-sends turns 1 to 9. Over a long chat the total bill grows roughly with the square of the number of turns.
- 3
What happens on overflow. If input plus the reply you allow exceeds the window, a good API returns an error (usually HTTP 400 with a message naming the limit, see Topic 2.6), and some services silently cut the oldest text. You want neither surprise. Count tokens before sending (Topic 1.2) and decide yourself what to drop. Always leave room for the reply: an input that fills the window leaves no space to answer.
- 4
Big window does not mean perfect memory. Models use text near the start and the end of a long input more reliably than text buried in the middle, an effect called lost in the middle. Long inputs also cost more and are slower. So put the most important instructions and facts at the start or the end, and send only what is relevant instead of everything that fits.
- 5
Trimming strategies. (1) Drop the oldest turns, always keeping the system prompt and the newest message (the example). (2) Summarise old turns into a short note and keep that instead. (3) Store facts outside the chat and retrieve only the relevant ones (retrieval, Phase 8). Dropping blindly can remove a question while keeping its answer, so many apps drop whole question and answer pairs, or summarise instead.
Explain it without notes
What is the context window, and what counts against it?
Why do chat apps cost more per turn as a conversation gets longer?
How would you handle a conversation that no longer fits the window?
What does lost in the middle mean for how you build prompts?
Practice
Write total_tokens(messages, count) that returns the sum of count(text) over (role, text) pairs. Use count = lambda t: len(t.split()) on two messages.
Write drop_oldest(messages, budget, count) that keeps the first message and removes the second, third... one by one until the total is at most budget (never removing the last). Return the kept list of texts.
Trade-offs
- ↔
Dropping old turns is simple but loses information. Summarising keeps the thread at the price of an extra call.
- ↔
A larger reply reserve allows longer answers and shrinks the room for context.
- ↔
Sending more context can improve answers and raises cost and latency, and may bury the key fact.
Done when you can
Done when you can explain that input plus output share the window.
Done when you can explain why the API is stateless and what the app must do.
Done when you can write a trimmer that keeps the system prompt and the newest message.
Done when you can name three strategies for long conversations.