Command Palette

Search for a command to run...

Hectal
PHASE 2Beginner ~21 min· topic 2 of 9

Topic 2.2

Messages and Roles

In one line

A chat is a list of messages, each with a role: system (rules), user (the person) and assistant (the model). The model remembers nothing between calls, so your program stores the list and sends all of it every time, trimming it when it grows too long.

0/9 · 0%

Think of it like this

Passing a notebook to a stranger. Each time you want help you hand over the whole notebook: the rules at the top, then every question and answer so far, then your new question. The stranger reads it, writes one answer and gives it back, then forgets everything. Next time you must hand over the notebook again, with the new lines added.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Message
One entry in the conversation list: a role and its content.
Role
Who is speaking: system (developer rules), user (the person) or assistant (the model).
System message
Standing instructions that apply to the whole conversation, usually the first message.
Conversation history
The list of earlier messages that your program stores and re-sends.
Stateless
Keeping no memory between calls. Model APIs are stateless; your app holds the state.
Context window
The maximum number of tokens the model can read in one call, input and output together.
Trimming
Removing old messages so the list fits a token budget.

Step by step

01Step 1: one message is a role and content

A message is a small object: {"role": "user", "content": "Hello"}. The request holds a list of them, in order. Order matters: the model reads them top to bottom.

02Step 2: the model forgets

Call the fake model with only What is my name?. It answers that it does not know. This is not a bug: nothing in its input contains a name. A real model behaves the same way (it may guess or make something up, which is worse).

03Step 3: keep a history list

The ask function appends the user's text to history, sends the whole list, then appends the model's reply as an assistant message. The next call sends everything again, so the name is now in the input and the answer is correct.

Step 3: keep a history listdiagram
Rendering diagram…

04Step 4: see what a forgetful app does

The example also sends only the last message. The answer is "I do not know your name" again. Bugs like this appear when a web server handles each request separately and nobody saves the history between them.

05Step 5: history costs money and space

Each turn adds two messages, and all of them are sent every time. After ten turns a chat may send twenty messages. The second example estimates tokens as characters divided by 4 plus a few tokens per message, a rough rule (Topic 1.2 explains real tokenizers).

06Step 6: trim from the oldest end

With a big budget nothing is dropped. As the budget shrinks, the oldest turns go first, the system message stays, and the list never begins with an assistant message. At the smallest budget only the system message and the newest question remain, which is the least that still makes sense.

Try it yourself

  1. 1

    Predict the forgetful app

    Before running the first example, predict each of the four answers (fresh call, turn 1, turn 2, forgetful app). Then run and check.

  2. 2

    Move the name

    In the first example change turn 1 to Hello there. (no name). What does turn 2 say now, and why?

  3. 3

    Find the breaking budget

    In the second example try budgets 100, 45, 30 and 10. At which budget does the list shrink to only the system message? What would you do if even the system message plus the newest question does not fit?

Code & diagrams

The model remembers nothing: your list does New tab

The fake finds the name only if it is in the messages it receives, which is how a real model behaves too.

Sign in to run this example in your browser.

Expected output

fresh call:    I do not know your name.
turn 1:        Your name is Asha.
turn 2:        Your name is Asha.
roles so far:  user, assistant, user, assistant
forgetful app: I do not know your name.   (sent only the last question)
lines in the model's input on turn 2: 3
Trimming a conversation to a token budget New tab

Token counts are estimates (4 characters per token plus overhead). Real counts come from the reply's usage (Topic 2.3).

Sign in to run this example in your browser.

Expected output

budget 200: kept 6 of 6 (76 tokens) -> system, user, assistant, user, assistant, user
budget  60: kept 4 of 6 (50 tokens) -> system, user, assistant, user
budget  40: kept 2 of 6 (22 tokens) -> system, user
budget  25: kept 2 of 6 (22 tokens) -> system, user

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget to store the reply

In the first example's ask, remove the line that appends the model's reply to history (the assistant message), then run it again.

terminal
$ Run
── what you'll see ──
fresh call: I do not know your name.
turn 1: Your name is Asha.
turn 2: Your name is Asha.
roles so far: user, user
forgetful app: Your name is Asha. (sent only the last question)
lines in the model's input on turn 2: 1

Myth vs fact

Myth

The model remembers my previous chat.

Fact

Not unless your app sends it again. Some products add memory features, but those are the app storing notes and putting them in the next request.

Myth

A longer conversation costs the same per message.

Fact

The whole history is input on every call, so later messages cost more and respond slower.

Myth

The system message cannot be overridden.

Fact

It carries more weight than user text, but it is not a security boundary. Users can still try to steer the model (Phase 10).

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

  • ▸

    Trim by tokens, not by message count: one pasted document can outweigh twenty chat lines. Count with the provider's usage from earlier calls where you can.

  • ▸

    Many providers cache a repeated prompt prefix at a discount (prompt caching). Keep the system message and early history byte-identical between calls so the prefix can be reused (Topic 6.6).

  • ▸

    Store the history on the server with a conversation id, not in the browser: clients can edit what they send, including fake assistant or system messages.

Remember this

  1. 1

    Why this exists. The model is a function (Topic 0.1): text in, text out, no memory. A conversation therefore has to live in your program. The API needs a standard way to say who wrote each line, so the input is a list of messages, each with a role and content. Without roles the model could not tell your rules from the user's words or from its own earlier answers.

  2. 2

    The three roles. system holds standing instructions from you, the developer ("Answer in one short sentence"); it normally comes first. user is what the person (or your program on their behalf) says. assistant is what the model said earlier; you add these back into the list yourself. Some APIs add more roles: tool for the result of a tool the model asked for (Phase 7), and newer OpenAI models also accept developer as the name for system-style instructions. Anthropic's Messages API takes the system text as a separate top-level system field and only user and assistant messages in the list (Topic 2.5).

  3. 3

    The model remembers nothing. Ask "What is my name?" in a fresh call and the model cannot know. If your app sends only the newest message, it forgets everything else. The first example shows three calls: a fresh one (does not know), a proper conversation (the name is in the list, so it answers), and a forgetful app that sent only the last message (does not know again). When a chat seems to remember, it is because the app re-sent the history.

  4. 4

    You pay for the history every turn. Turn 1 sends 1 message, turn 2 sends 3, turn 3 sends 5. Input tokens grow with every turn, so a long chat costs more per message than a short one, and it gets slower. At some point the list also stops fitting in the model's context window (the most tokens it can read at once, Topic 1.5).

  5. 5

    Trimming. The simplest fix is to drop the oldest turns. The second example keeps the system message (it holds the rules), removes the oldest messages until an estimated token count fits a budget, and never leaves an assistant message first (its question was dropped, so it would be confusing). Better fixes exist: summarise old turns, but the list you send is always something your code builds on purpose.

  6. 6

    Messages are data, so treat user text carefully. A user can type text that looks like instructions. Never paste user text into the system message; put it in a user message, and use delimiters (Topic 3.4). The role is the first line of defence, not the last (Phase 10).

Explain it without notes

01

Why must an application re-send the conversation on every call?

02

What are the roles in a chat request and who writes each?

03

How would you stop a long conversation from overflowing the context window?

Practice

01

Write add_turn(history, user_text, reply_text) that appends a user and an assistant message to history and returns it.

02

Write last_user_message(messages) that returns the content of the newest user message, or None if there is none.

Trade-offs

  • ↔

    Sending full history gives the best continuity but costs more and slows down; trimming or summarising saves money and can lose details.

  • ↔

    Server-side history is safer and can be shared across devices; client-side history is simpler but can be tampered with.

Done when you can

  • Done when you can explain why a chat app re-sends the whole conversation.

  • Done when you can name the roles and who writes each one.

  • Done when you can keep a history list and add each reply to it.

  • Done when you can trim a history to a token budget without dropping the system message.