Topic 14.2
System Design: A Customer-Support Assistant
In one line
Design a support assistant that answers from help articles, looks up orders and hands over to a human, with numbers for traffic, tokens, cost and storage. You will practise the interview flow: requirements, estimates, architecture, safety, evaluation, operations.
Think of it like this
Opening a help desk in a shop. Before hiring anyone you ask how many customers come in an hour, what they ask, which questions only a manager may answer, what the desk must never promise, and how you will know whether customers leave happy. The interview is the same planning, done on a whiteboard in 45 minutes.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Requirements
- What the system must do and the limits it must stay inside, written with numbers where possible.
- Back-of-envelope estimate
- A quick calculation with round numbers to see whether a design is possible and roughly what it costs.
- Peak factor
- How much busier the busiest minute is than the daily average, such as 5 times.
- Little's law
- Calls in flight = arrival rate x time per call. It tells you how many calls run at once.
- Human hand-off
- Passing the conversation to a person when the bot should not or cannot continue.
- Shadow mode
- Running the new system on real traffic without showing its answers, to compare it with the old one safely.
- p95
- The time that 95 out of 100 requests beat. It shows the slow tail that an average hides. See Topic 6.3.
- Golden set
- A saved list of real inputs with the answers you expect, used to score every change. See Topic 9.2.
- Guard
- A check in plain code before or after the model, such as redacting personal data or checking that a citation exists. See Topic 10.9.
Step by step
01Step 1: ask questions and fix the requirements
Ask who the users are, which questions are common, which actions the bot may take, which languages, and what failure costs. Then write numbers on the board: 50,000 conversations a day, about 6 turns each, first token within 1.5 seconds at p95, a monthly budget, and a rule list (never promise refunds, always cite a help article, hand over when unsure).
02Step 2: estimate load, tokens and cost
Calls per day = 50,000 x 6 = 300,000, which is about 3.5 per second on average and about 17 at a peak factor of 5. With three seconds per call, Little's law gives about 53 calls in flight. Input tokens per day come to about 888 million and output tokens to 54 million. Run the estimator and read the cost lines.
03Step 3: draw the main flow
Draw the request path from the customer to the answer and name each box's job. Keep the first drawing small, then add the guards and the hand-off. The diagram shows one reasonable layout.
04Step 4: go deep where the risk is
Retrieval: chunk articles by heading, keep title and date as metadata, search by keyword and vectors together, pass the best three to five chunks with ids, and answer "I do not know" when nothing relevant is found. Tools: order lookup is read-only; anything that changes money is a suggestion to a human, never an action.
05Step 5: say how you will evaluate it
Build a golden set from real past tickets, split by topic, including cases where the right answer is "hand over". Score correctness, groundedness (every claim supported by a cited chunk) and policy. Gate every prompt or model change in CI, then watch thumbs, escalation rate and sampled conversations in production.
06Step 6: operations and rollout
Dashboards for p95 latency, time to first token, cost per conversation, fallback rate and escalation rate. Alerts on cost spikes and error bursts. Roll out in shadow mode first, then 5% of traffic, then more, with a kill switch that routes everyone to a human queue. Store ids of retrieved chunks, not their full text, to keep storage and privacy risk small.
Try it yourself
- 1
Change a requirement
In the first estimator, change
TURNSfrom 6 to 10 and run it. By what factor did the input tokens grow, and why is it more than 10/6? - 2
Pull a cost lever
Change the what-if from 1000 retrieved tokens to 600. Then lower
PRICE_INto a cheaper-model example price and see the combined effect. Which lever saved more? - 3
Try the mock interview
Send the lab, answer the question it asks with the six-step order from this topic in about 150 words, and send again. Did the feedback name a gap you also see in your own answer?
Code & diagrams
Example prices and traffic. The loop re-sends earlier turns, which is why input tokens dominate.
Expected output
tokens per day: 888000000 in, 54000000 out
cost per day: 525 dollars, per month: 15750 dollars
requests per second: average 3.47, peak 17.36
calls in flight at peak: 53
input tokens per minute at peak: 3081360
what-if, 1000 retrieved tokens instead of 1500: saves 2250 dollars per monthBytes per token, row overhead and retention are assumptions you say out loud in the interview.
Expected output
messages per day: 600000
stored per day: 0.5 GB (messages 0.4, turn logs 0.1)
kept 90 days: 44.8 GB, with 3 replicas: 134.5 GB
if every turn also stored its retrieved text: 206.8 GB for 90 days
a page of 50 messages read back: about 34 KBTry: Send the request. The model plays an interviewer and asks one LLM system design question. Edit the user message to give your answer in four or five sentences using the order from this topic, send again, and read the feedback and the follow-up question.
Goals
- The reply contains a question
- The question is short (under 120 words)
- The reply finished on its own
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget that history is re-sent
In estimate, set history = 0 (Python) so earlier turns are not counted.
Myth vs fact
Myth
A system design answer is a picture of boxes.
Fact
The boxes are the easy part. Interviewers score the numbers, the trade-offs, the safety plan and the evaluation plan.
Myth
Use the biggest model for the best quality.
Fact
Quality is measured on your tasks. A routed design (small model for easy turns, larger for hard ones) is usually cheaper and fast enough, and your eval set shows where it is safe.
Myth
The bot should always answer.
Fact
Saying "I do not know, here is a person" is often the correct and cheaper outcome. Count hand-overs as a feature.
Interview problem
The problem
A customer-support assistant for an online shop
Design an assistant that answers customer questions in a chat window using the shop's help articles, can look up an order's status, and hands over to a human when it should. It must be fast, affordable and safe.
You're given
- 50,000 conversations a day, about 6 turns each (300,000 model calls a day).
- First token within 1.5 seconds and full reply within 6 seconds at p95.
- Monthly model budget around 15,000 dollars at example prices (check your provider).
- Answers must come from the help articles, with the article id cited. Never promise a refund or change an account.
- Personal data must not reach logs or third parties unredacted.
- A human can take over at any time, and the bot hands over on its own when unsure.
The interviewer follows up
Cost doubled overnight. What do you check first?
A customer types an instruction telling the bot to approve a refund. What happens?
How do you know a new prompt is better?
The provider starts returning 429 for every call. What do you do?
Black Friday brings 10 times the traffic. What changes?
When it breaks
The model answers a refund question from memory because retrieval found nothing
What you see
A confident wrong policy; the customer expects money back.
Fix & prevent
Refuse when no chunk passes a relevance threshold, test with no-answer questions, and check citations in the output guard.
History is re-sent in full on every turn and never trimmed
What you see
Input tokens and latency grow with each turn; long chats hit the context limit.
Fix & prevent
Trim to a token budget that keeps the system message (Topic 14.4) and summarise old turns.
Full retrieved text and customer messages are logged unredacted
What you see
Personal data sits in logs and dashboards for months.
Fix & prevent
Redact before logging, store chunk ids instead of text and set a retention period.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cost lever order: shorter retrieved text and capped output first (no quality change if tested), then caches and prompt caching for the stable prefix, then routing to a cheaper model for easy turns, and only then a smaller prompt that risks quality. Always re-run the eval after each change.
- ▸
Rate limits are per minute in tokens and requests. At peak the example needs about three million input tokens a minute, which is a number to compare with the provider's tier before you promise the design.
- ▸
Say what the estimate leaves out: it prices only the model calls. Retrieval infrastructure, the human agents who take the hand-offs, evaluation runs and the tokens of retries are extra. If the bot uses a reasoning model, its hidden reasoning tokens are billed as output and would push the output side well above the 180 tokens per reply assumed here (Topic 1.8).
- ▸
Provider prompt caching helps only with a stable prefix. Here that is the 800-token system message, not the 1,500 retrieved tokens, which change on every turn, so do not claim a saving on the whole input (Topic 6.6).
- ▸
Keep each conversation id across logs, traces and the order tool call, so one customer complaint can be followed through every layer (Topic 11.3).
Remember this
- 1
Why this exists. A system design interview checks whether you can turn a vague product idea into something that survives real traffic, real costs and real attackers. For LLM systems the usual traps are leaving out numbers, drawing one box labelled "LLM", and forgetting safety, evaluation and failure handling. The answer is a clear order of work that you repeat every time.
- 2
The order that works. (1) Ask questions and write down requirements with numbers. (2) Estimate load, tokens and cost on the board. (3) Draw the main flow, then name each box's job. (4) Go deep where the risk is: retrieval, tools, safety. (5) Say how you will evaluate it before and after launch. (6) Cover operations and failure: limits, fallbacks, logs, alerts, rollout. Say the order out loud at the start, and the interviewer will help you keep to it.
- 3
Estimates are the credibility test. Tokens per day, cost per month, peak requests per second and storage do not need to be exact. They need correct arithmetic and visible assumptions. The runnable estimators below use 50,000 conversations a day with six turns each. Notice that each turn re-sends the earlier turns, so input tokens grow faster than the number of turns.
- 4
Boxes with jobs. A gateway authenticates and rate-limits. An input guard redacts personal data and checks for injection. A router decides between a help-article answer, an order lookup and a human. Retrieval finds help-article chunks. A read-only order tool fetches facts. A model client handles timeouts, retries, fallback and caching. An output guard checks citations and policy. A log and metrics path sees every step.
- 5
Never promise what only a human may. A support bot must not promise refunds, change accounts without confirmation, or give legal or medical advice. Draw the human hand-off as a first-class box with clear triggers: angry customer, money, no evidence found, repeated failure. A bot that knows when to stop is safer and often has a better resolution rate than one that always answers.
- 6
Show the cost lever. The first estimate often lands just over budget. Say which lever you would pull and by how much: fewer retrieved tokens, a cache, a small model for easy turns, provider prompt caching for the stable system prefix. The what-if line in the estimator shows how cutting 500 retrieved tokens per turn saves 2,250 dollars a month (500 tokens x 6 turns x 50,000 conversations x 30 days at 0.50 dollars per million).
Explain it without notes
Why do you start a system design answer with questions and numbers instead of drawing boxes?
Why do input tokens grow faster than the number of turns in a chat, and how do you control it?
Which parts of the support bot are deterministic code, and which are the model, and why does the split matter?
How would you roll out the assistant safely?
Practice
A shop's bot gets 30,000 calls a day, the busiest minute is 4 times the average, and each call takes 2.5 seconds. Print the average and peak requests per second and how many calls are in flight at the peak.
Each stored message averages 150 tokens at 4 bytes per token plus 200 bytes of row overhead. For 1 million messages a day kept 30 days, print the storage per day and in total in gigabytes.
Trade-offs
- ↔
Keeping full retrieved text per turn makes audits easy but multiplies storage and privacy risk; keep chunk ids and the document version instead.
- ↔
A cache cuts cost and latency but can serve a stale policy; give entries a short lifetime and clear them when an article changes.
- ↔
More hand-overs lower risk but raise human cost; tune the thresholds with the golden set and the escalation dashboard.
Done when you can
Done when you can open an answer with questions and written numbers.
Done when you can compute tokens per day, cost per month, peak requests per second and calls in flight.
Done when you can draw the flow and say each box's job in one sentence.
Done when you can state three safety rules and where code enforces them.
Done when you can describe the evaluation gate and the rollout steps.