PHASE 4Intermediate ~30 min· topic 3 of 8

Topic 4.3

Sending History and the Context Window

In one line

A model remembers nothing between requests. Its "memory" is the conversation your client resends every time, so each request grows, costs more, and eventually hits the context window. You trim or summarise the old turns, and tell the user when the model can no longer see them.

Think of it like this

Phoning a consultant who loses their memory after every call. To get a useful answer you must start each call by reading out the whole case file so far, then ask your new question. The longer the file, the longer each call takes and the more it costs. And the phone line has a maximum length: past it, the oldest pages can't be read out at all. A chat with a model works exactly like this, and your client is the one reading out the file.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Stateless: Keeping nothing between requests. The model only knows what is inside the current request.
History: The earlier messages of the chat, resent with every new request.
Context window: The most tokens one request can hold, input and reply together. It differs by model.
Input tokens: The tokens you send: the whole history and your instructions. You pay for them each time.
Output tokens: The tokens the model writes in its reply.
Usage: The token counts the provider returns with a response (prompt_tokens, completion_tokens).
Sliding window: Keeping only the newest messages that fit in the budget.
Summary turn: A short message that replaces dropped messages by saying what they contained.
Token estimate: A cheap guess of token count from text length, used to decide what to trim.

Step by step

01Two requests, no memory between them

These are the whole bodies of two requests. The second one only works because it repeats the first. Nothing on the provider's side connects them.

two requestswhole filejs

// request 1
{ messages: [ { role: "user", content: "My name is Asha" } ] }

// request 2: the model knows the name only because turn 1 is inside it
{ messages: [
    { role: "user", content: "My name is Asha" },
    { role: "assistant", content: "Nice to meet you!" },
    { role: "user", content: "What is my name?" },
] }

02Prove it, and watch the request grow

The first live example runs three turns twice. Client A resends the whole history; client B sends only the newest message. The stand-in model reads its messages and answers from them, so what you see is exactly what arrived.

Look at the "sent" lines: 58 characters, then 188, then 307. Each request carries everything before it. Client B's answer to "What is my name?" is "I don't know your name yet.", the exact complaint users make when a client forgets to resend.

The last part sends an empty history, as a buggy trimming function might. The server answers 400 and the client must show an error, not a blank screen.

03Estimate, then trim

The second live example estimates tokens (4 characters each, plus 4 per message for the wrapper) and walks backwards from the newest message, keeping what fits in 60 tokens. Read the turn lines: while the history is small nothing is dropped; from turn 3 on, the oldest messages fall off.

Then it shows the two edges: one message far over the budget is still sent (a provider error beats sending nothing), and a summary message is put in front of what is kept.

Estimate, then trimdiagram

Rendering diagram…

04Say it on the screen

The third live example adds a notice that appears the moment a message falls out of the budget, and greys the dropped messages. The count in the notice comes from the same function that decides what to send, so the screen can never claim something different from what was sent.

05Correct the guess with the real number

Every response reports usage.prompt_tokens. If your estimate said 800 and the provider says 1,100, your estimate is low for this chat (maybe it is mostly Hindi or code). Scale your next estimates by that ratio, or lower the budget, instead of waiting for a 400.

Try it yourself

1
Make the budget bigger
In the second example change the 60 in the trimHistory(history, 60) call inside the turns.forEach loop to 200 and run it. Every turn now says "dropped 0", and the "sent" token count grows each turn (16, 49, 79, 108, 137).
That growing number is what you pay for on every request. Find the turn where it would have crossed a context window of, say, 4,000 tokens if the chat kept going.
2
Count Hindi
Add console.log(estimate("चाय मिलेगी? ☕"), estimate("Tea is available?")) at the end of the second example. It prints 4 5: the estimate counts characters, so the shorter Hindi line looks cheaper than the English one.
Real tokenizers usually split non-Latin text and emoji into more tokens per character, so the estimate is low for them. A product with many Hindi users should leave a bigger safety margin, or measure with the provider's usage numbers.
3
Drop the question but keep the answer
In the second example, add this line after the turns.forEach(...) block: const bad = history.slice(1); console.log(bad[0].role);.
It prints assistant: a naive cut can leave a history that begins with the model's own words. Some providers reject that or answer oddly. Trimming in whole question-and-answer pairs avoids it.

Code & diagrams

The model only knows what you resend

The stand-in model answers from the messages it receives, so the replies prove what the client sent. In a real app the one line that changes is the server route: it calls the provider instead of the stand-in. The model code is not what makes the difference here; the request body is.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

A: resend everything
  sent 1 message(s), 58 characters
  My name is Asha -> Nice to meet you. I received 1 message.
  sent 3 message(s), 188 characters
  Do you sell ginger biscuits? -> Nice to meet you. I received 3 messages.
  sent 5 message(s), 307 characters
  What is my name? -> Your name is Asha.
B: send only the newest message
  sent 1 message(s), 58 characters
  My name is Asha -> Nice to meet you. I received 1 message.
  sent 1 message(s), 71 characters
  Do you sell ginger biscuits? -> Nice to meet you. I received 1 message.
  sent 1 message(s), 59 characters
  What is my name? -> I don't know your name yet.
C: empty history
  HTTP 400 'messages' is required and must not be empty.

Trimming to a budget, and the edge cases

The token numbers are estimates (length divided by 4, plus 4 per message): good enough to decide what to drop, never to bill anyone. The summary line is built by hand here; a real one comes from a model.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

budget: 60 tokens
turn 1: history 1 messages (16 tokens) -> sent 1 (16 tokens), dropped 0
turn 2: history 3 messages (49 tokens) -> sent 3 (49 tokens), dropped 0
turn 3: history 5 messages (79 tokens) -> sent 3 (49 tokens), dropped 2
turn 4: history 7 messages (108 tokens) -> sent 4 (59 tokens), dropped 3
turn 5: history 9 messages (137 tokens) -> sent 4 (58 tokens), dropped 5
one huge message: 1 kept, 504 tokens used of 60
with a summary: system, user, assistant, user, assistant
Earlier in this chat (6 messages) the customer said: Hi, I want to order masala chai for my office.

Telling the user what the model can no longer see

Greyed messages are still on screen but no longer sent. The notice and the grey come from the same count as the request, so they cannot disagree with it. role="status" lets screen readers announce the notice politely.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

5 messages: Mitra can no longer see the first 1 message of this chat.
6 messages: Mitra can no longer see the first 2 messages of this chat.
7 messages: Mitra can no longer see the first 3 messages of this chat. | greyed: 3

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Let the newest message be dropped

In the second example, change if (kept.length > 0 && used + c > budget) break; to if (used + c > budget) break;.

terminal

$ Console

── what you'll see ──

one huge message: 0 kept, 0 tokens used of 60

Myth vs fact

Myth

The model remembers the conversation.

Fact

It remembers nothing. Every request must contain whatever you want it to know. What looks like memory is your client resending the history.

Myth

A bigger context window means I can stop trimming.

Fact

A bigger window only moves the limit. Requests still get slower and more expensive with every turn, and very long inputs are often used less reliably by the model.

Myth

Estimating tokens as length divided by 4 is accurate.

Fact

It is a rough guide for English prose. Other scripts, emoji and code take more tokens per character. Use the provider's usage for the real number.

Myth

Trimming silently is fine, since the model still answers.

Fact

The model answers, but it may now contradict the earlier chat. The user must be told that older messages are no longer seen.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep the system prompt out of the trimming. It is always sent first and always counts toward the budget. Subtract its size before you walk the messages.
▸
Trim at turn boundaries, and never drop a message that a later one depends on (an assistant message that contains a tool call and the tool's result are a pair, Phase 8). A history with a result but no call is rejected by providers.
▸
Do the summary in the background after the chat grows past a threshold, and store it as its own message so you don't pay for it again. Make the summary request cheap: it sees only the part being dropped.
▸
Log usage.prompt_tokens per chat. A chat whose prompt tokens keep climbing is your most expensive user, and the data to decide a limit.

Remember this

1
The problem this solves: The screen shows a long conversation, so it feels as if Mitra remembers it. It doesn't. If your code sends only the newest question, Mitra answers "I don't know your name" right after being told it. And if it sends everything forever, the request gets slower, dearer, and finally fails. This topic shows what to send, how it grows, and what to do when it no longer fits.
2
Models are stateless. *(Recall from Topic 4.1: toApi keeps only the done messages and strips them to role and content.)* Each request is independent: the model reads the messages in that one request and writes a reply, then forgets it all. The only way it can "remember" turn 1 during turn 6 is that turn 1 is *in the request for turn 6*. So on every send your client builds the list of messages (Topic 4.1's toApi) and sends all of it. The first live example proves it: with the full history the stand-in model answers "Your name is Asha.", and with only the newest message it can't.
3
Cost grows with every turn. You pay for input tokens (everything you send) and output tokens (the reply). Turn 1 sends one message. Turn 10 sends nineteen, all of which you paid for again. Over a whole conversation the input you pay for grows roughly with the *square* of the number of turns, not linearly. Long chats are the expensive ones, which is why providers offer cheaper prices for repeated (cached) prefixes (Topic 13.4).
4
The context window is the most tokens a model accepts in one request, counting your input *and* the reply it writes. Each model has its own limit; read it from your provider's documentation, never from memory, because it changes between models and over time. If a request is over the limit the provider answers with an error (a 400 with a message about the length, whose exact wording differs by provider), and the user sees a failure at exactly the moment the chat got long and interesting.
5
Counting tokens. You can't know the exact count in the browser without a tokenizer library, but you need a number to decide what to drop. A rough rule for English is about 4 characters per token. Hindi, other non-Latin scripts, emoji and code usually take *more* tokens per character, so the rule undercounts them: leave a safety margin. The best real number is the one the provider reports: usage.prompt_tokens in each response (Topic 1.1). Use the estimate before sending and the real count afterwards to correct it.
6
Trimming: a sliding window. Walk from the newest message backwards and keep messages until the budget is used up; drop the rest. Rules that keep it safe: always keep the newest message (if even that is over the budget, send it anyway and let the provider say so, rather than sending nothing); drop in whole turns where you can (a question without its answer, or the other way round, confuses the model, and some providers require roles to alternate or the first message to be from the user, so check yours); and leave room for the reply: the budget is the window minus the longest answer you allow, minus your system prompt.
7
A summary turn. Instead of throwing old messages away, replace them with one short message that says what happened ("Earlier the customer said they are ordering masala chai for an office of ten"). The summary is normally written by a model in a separate request, so it costs a little and can lose details or get them wrong. It keeps the thread of a long chat; it is not a perfect memory. The model side of memory, summarising and retrieval is taught in /llm-engineering and /rag; here you only need to know it changes what your client sends.
8
Tell the user. The screen still shows all the messages, so the user believes Mitra sees them all. When something is dropped, say so on the screen ("Mitra can no longer see the first 3 messages of this chat"), and grey those messages. Otherwise the first time the model "forgets" something, the user thinks the product is broken. Offer a way out: start a new chat, or pin the facts that matter.
9
On the server, enforce limits too. The browser decides what to send, but never trust it: your route must cap the number of messages and the size of the body (Topic 1.5), or one client can send a million-token history on your bill.

Explain it without notes

Why does a chat feel like the model remembers, when it doesn't?

Why do long conversations cost more per message than short ones?

How would you trim a history safely, and what must you always keep?

What should the user see when old messages are dropped, and why?

Practice

Write trimPairs(messages, budget) that keeps whole user+assistant pairs from the newest end, using the estimate function from the example, and never returns an empty list.

A response contains usage: { prompt_tokens: 1100 } but your estimate for that request was 800. Write a function calibrate(estimated, actual) that returns the factor to multiply future estimates by, and show how to use it in trimHistory.

Trade-offs

↔
Sending the full history gives the best answers and the highest cost. A sliding window caps the cost and the latency but loses old facts. A summary keeps the thread cheaply but can distort it. Most products start with a window and add summaries only when users complain about forgotten context.
↔
A generous budget wastes money; a tight one makes Mitra forget. Set it from your longest realistic chat and your provider's price, then watch usage in production before changing it.

Done when you can

I can explain why a model needs the history resent and what happens if I don't.
I can estimate tokens and say when the estimate is wrong.
I can trim a history to a budget and keep the newest message.
I can show the user which messages the model no longer sees.
I can describe what a summary turn costs and what it can lose.

Command Palette

Words you'll meet

Step by step

01Two requests, no memory between them

02Prove it, and watch the request grow

03Estimate, then trim

04Say it on the screen

05Correct the guess with the real number

Try it yourself

Make the budget bigger

Count Hindi

Drop the question but keep the answer

Code & diagrams

Break it on purpose

Let the newest message be dropped

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can