Topic 12.3
Storing Conversations
In one line
The model remembers nothing, so your service must store each conversation and send the right part of it every time. That means a database table, a token budget that trims old messages, an optional summary of what was trimmed, strict separation between users, and a way to delete data on request.
Think of it like this
A doctor's notes. The doctor does not remember every patient, so each visit starts by reading the file. A long file is too much to read in a short appointment, so the doctor reads the summary at the top and the last few visits in full. And each file belongs to one patient: handing over the wrong one is a serious mistake, and a patient can ask for theirs to be erased.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Conversation
- A stored list of messages that belong together and to one owner.
- Token budget
- The most tokens you allow yourself to send as context for one call.
- Summary
- A short text that stands in for older messages that no longer fit.
- Sliding window
- Sending only the newest messages that fit the budget; older ones slide out.
- Retention
- How long stored data is kept before it is deleted.
- Row
- One record in a database table, such as one message.
Step by step
01Step 1: a list in memory
The demo version is history = []. It works for one user in one process and fails on the first restart, the second running copy, or the second user.
02Step 2: two tables
One row per conversation (with its owner and summary) and one row per message (with a token count). Every read filters by the owner. The index on (conversation_id, id) makes "the newest messages of this conversation" fast.
CREATE TABLE conversations (
id BIGSERIAL PRIMARY KEY,
tenant_id TEXT NOT NULL,
user_id TEXT NOT NULL,
summary TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE messages (
id BIGSERIAL PRIMARY KEY,
conversation_id BIGINT NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
role TEXT NOT NULL CHECK (role IN ('system', 'user', 'assistant', 'tool')),
content TEXT NOT NULL,
tokens INT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON messages (conversation_id, id);03Step 3: trim by budget
The example store counts words as stand-in tokens. Each add appends the message, and while the total (summary plus messages) is over the budget it removes the oldest message. The first three turns reach a total of 14, which still fits. The fourth turn pushes it over, so old messages start to leave.
04Step 4: leave a trace in the summary
Every removed message adds its first three words to the summary, which is capped at six words. It is a toy; a real summariser would be a model call that writes one or two sentences. The shape is the same: older text becomes a short string that is sent first.
05Step 5: separate users and support deletion
Ben has his own conversation: asking for his context returns one line and does not touch Ana's. forget removes a user's data and returns whether anything was there. The second call returns no, which is how you make deletion safe to repeat.
06Step 6: put the check in the query, not in the caller
A real store should make the owner part of every function: get_context(tenant_id, user_id, conversation_id), and the SQL has WHERE id = $1 AND tenant_id = $2 AND user_id = $3. Then a caller cannot forget the check.
Try it yourself
- 1
Predict the trim
With a budget of 14, which message leaves first, and what does the summary contain after the fifth turn? Work it out by hand, then run.
- 2
Change the budget
Set the budget to 30 and run. How many messages are kept and is there a summary at all?
- 3
Protect the system prompt
Add a system message first in the conversation and change
addso that a message with the rolesystemis never removed. What must the loop do instead when the oldest message is a system message?
Code & diagrams
Words stand in for tokens and a few kept words stand in for a model-written summary.
Expected output
after user: messages=1 total=6
after assistant: messages=2 total=12
after user: messages=2 total=14
after assistant: messages=1 total=14
after user: messages=1 total=12
context sent to the model:
[summary] still dropping after then update the
[user] firmware updated and it works now
ben sees 1 line, ana sees 2
forget ana: yes, again: noBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trim from the wrong end
In add, remove the newest message instead of the oldest: use conv["messages"].pop() (JavaScript: conv.messages.pop(), Java: remove(conv.messages.size() - 1)).
Myth vs fact
Myth
A bigger context window means I do not need to manage history.
Fact
Every resent token costs money and time, and quality can drop in very long contexts (Topic 1.5). A budget keeps cost and latency predictable.
Myth
A summary is as good as the original messages.
Fact
It loses detail and can add mistakes. Keep the original messages stored; send the summary, not delete the truth.
Myth
Checking the user in the web page is enough to protect conversations.
Fact
The check belongs in the server and in the query. A caller can change an id in a request by hand.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Summarise in the background when a conversation crosses a threshold (for example 70 percent of the budget), not in the request path, and store which message the summary covers up to so it can be re-run.
- ▸
Prompt caching rewards stable prefixes (Topic 6.6). A summary that changes on every turn defeats it; update the summary in larger steps, not after each message.
- ▸
Encrypt sensitive stored text at rest, restrict who can read raw conversations, and give support staff redacted views. Deletion must also cover backups' retention and any search or analytics copies.
Remember this
- 1
Why this exists. The API is stateless (Topic 1.5 and 2.2): every request must contain the whole conversation you want the model to see. In a script that is a list in memory. In a service the list must survive restarts, work when the next request lands on a different running copy (Topic 12.1), and stay separate per user. So conversations become rows in storage.
- 2
What to store. For each message: its conversation, role, content, a token count, the time, and useful metadata such as the model and the prompt version that produced it (Topic 3.8). For each conversation: its owner (tenant and user), a title, a summary, and timestamps. Store the token count when the message is saved so trimming does not need to recount (Topic 1.2).
- 3
Trimming to a budget. The context window is limited, and cost grows with every token you resend (Topic 6.1). Build the context for each call from the newest messages backwards until a token budget is full, always keeping the system prompt, and make sure the latest user message is in. A budget is better than "the last 10 messages", because one pasted document can be thousands of tokens.
- 4
The summary slot. Throwing old messages away loses facts ("my router model is X"). Instead, when messages leave the window, fold them into a short summary stored on the conversation and sent as the first line of context. A real summary is produced by a model call (cheap model, background job, Topic 12.4); the example fakes it by keeping a few words so you can see the mechanics. Summaries lose detail and can introduce errors, so keep the original messages in storage even when they are no longer sent.
- 5
Per-user memory. Some products also keep durable facts across conversations ("prefers metric units"). Make this opt-in, visible and editable by the user, store it as separate small records, and add only the relevant facts to the prompt. Treat what is stored as untrusted text: a stored sentence is a way for an earlier injection to come back (Phase 10).
- 6
Privacy. Key every query by the owner: reading conversation 42 without checking that it belongs to the caller is the classic data leak (Topic 12.5). Decide how long data is kept, support deleting a user's data on request (the example's
forget), keep personal data out of ordinary logs (Topics 10.5 and 11.2), and tell users what is stored. Check what your model provider does with the data you send (Topic 10.4). - 7
Concurrency. Two browser tabs can send at the same time. Append messages with a database insert (never read-modify-write a whole list), order by an increasing id, and decide what happens when a second request arrives while the first reply is still being generated: reject it, queue it, or allow both. See the System Design course (/system-design) on transactions and concurrency.
Explain it without notes
Why must an LLM service store conversations itself?
How do you decide which messages to send to the model on each call?
What are the risks of summarising old messages, and how do you limit them?
How do you keep one user's conversation from leaking to another user?
Practice
Write build_context(messages, budget) where messages is a list of (role, text) and the budget counts words. Keep the newest messages that fit, in original order, and always keep the last message even if it alone is over the budget. Test it with a budget of 6.
Write can_read(conversation, user) that returns True only if conversation["owner"] == user, and show it blocks a different user.
Trade-offs
- ↔
Smaller budgets cost less and answer faster but forget more; bigger budgets remember more and cost more on every call.
- ↔
Summaries save tokens at the price of detail and an extra model call.
- ↔
Durable per-user memory improves the product and increases privacy duties.
Done when you can
Done when you can design the two tables for conversations and messages.
Done when you can build a context from the newest messages within a token budget.
Done when you can explain what a summary slot is for and its risks.
Done when you can show how a query makes cross-user reads impossible.
Done when you can implement and explain deletion of a user's data.