Topic 13.3
Design: A Customer-Support Assistant
In one line
A worked design for a multi-tenant support assistant: fresh tickets and articles, per-customer separation, hand-off to a human, and cost limits, with the hand-off and budget rules written and run.
Think of it like this
A shop assistant who answers what they know, calls a manager when unsure, and never opens another customer's file. A support assistant needs those same three habits: answer from the shop's own knowledge, hand over cleanly, and keep customers apart.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Tenant
- One customer organisation using the service, whose data must stay separate from the others'.
- Containment rate
- The share of conversations solved without a human.
- Hand-off
- Passing the conversation to a person, with the context so far.
- Change feed
- A stream of 'this record changed' events that keeps an index fresh.
- Daily budget
- A cap on money or tokens a tenant may use in a day.
- Never-automate list
- Topics the assistant must always hand to a person.
Step by step
01Step 1: separate public and private text
Help articles are public, tickets are per tenant. Give both a tenant field (public articles use a shared value) and filter on it in every search.
02Step 2: choose how to isolate tenants
A shared index with a mandatory filter is cheaper and simpler for many small tenants. A separate index per tenant is stronger isolation for a few large ones. State which and why.
03Step 3: keep it fresh
A change feed updates tickets within minutes. Articles sync on a schedule. Every chunk stores an updated-at time that appears in the answer as 'as of'.
04Step 4: write the hand-off rules
Order them strongest first: the customer asked for a person, the budget is used up, then the evidence is weak. The second weak answer in a row hands off instead of refusing again.
05Step 5: meter the cost
Count tokens per request against the tenant's daily limit. Record the cost so spend is visible per tenant, and alert on jumps.
06Step 6: measure containment honestly
Containment must be paired with wrong-answer rate on a human-read sample. Otherwise the metric rewards confident mistakes.
Try it yourself
- 1
Reorder the rules
Move the budget rule below the evidence rule and run. Which turn changes? Argue which order is right for a customer who needs help and a company that pays the bill.
- 2
Add a never-automate topic
Add an argument
topicand hand off whenever it equals 'refund over limit'. Add a turn that tests it. - 3
Raise the floor
Set MIN_EVIDENCE to 0.80 and run. How many turns now refuse or hand off? What happens to containment?
Code & diagrams
MIN_EVIDENCE of 0.30 is an example for a real embedding model's cosine. A real value comes from the golden set (Topic 8.4).
Expected output
clear question (best 0.82): answer with sources
weak match, first time (best 0.18): refuse and ask to rephrase
weak match again (best 0.21): hand off: second weak answer in a row
asks for a person (best 0.75): hand off: the customer asked for a person
budget used up (best 0.80): hand off: this tenant's daily budget is used up
weak match and budget used up (best 0.10): hand off: this tenant's daily budget is used upBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Never hand off on a weak answer
In the Python tab, change if refusals_in_row >= 1 to if False, so a weak answer always just asks to rephrase.
Myth vs fact
Myth
Higher containment is always better.
Fact
Containment counts conversations that did not reach a person, including those the assistant got wrong. Pair it with a sampled wrong-answer rate.
Myth
Tenant separation is solved by telling the model not to mention other customers.
Fact
The text must never be in the prompt. Separation happens in the search filter, the caches and the logs.
Myth
A hand-off means the assistant failed.
Fact
A clean hand-off with context is the correct outcome when the assistant is unsure. A wrong confident answer is the failure.
Interview problem
The problem
A multi-tenant customer-support assistant
Design an assistant for a software company's customers. It answers from public help articles and from the customer's own tickets, hands over to support staff when needed, and must not cost more than a fixed amount per customer per day.
You're given
- 2,000 customer organisations (tenants). Public articles shared, tickets private to each tenant.
- Tickets searchable within 5 minutes of creation. Articles may be an hour old.
- Hand off to a human when the customer asks, when evidence is weak twice, or on refunds above a limit.
- A daily token budget per tenant. Never show one tenant another's data.
- Measure whether it helps: containment, wrong answers, ratings.
The interviewer follows up
A customer reports seeing another company's ticket text. What do you do?
Support staff say the assistant is confidently wrong about a new feature. Why?
Costs double after a release. What do you check?
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cache keys must contain the tenant id. A shared answer cache keyed on the question text alone returns one customer's answer to another, a leak that no retrieval filter can prevent (Topics 10.4 and 11.3).
- ▸
Tickets contain personal data. Redact before indexing and before logging, set retention, and make deletion reach vectors and caches (Topics 10.5 and 10.6).
- ▸
Protect the hand-off path as well: during a model outage, the fallback is a human queue with a message, not an error page.
Remember this
- 1
Why this design differs from document Q&A. The corpus is two kinds of text with different rules: stable help-centre articles (public) and tickets or account notes (private, per customer). Answers must be fresh to the hour, the user is often upset, a wrong answer can cost money, and every conversation has a price.
- 2
Separation of tenants. Each customer company is a tenant. Every chunk carries a tenant id, and every query filters on it before ranking. Help articles carry a 'public' label that every tenant may see. Choose between one index per tenant and one shared index with a filter (Topic 10.4): few large tenants favour separate indexes, thousands of small ones favour a filter. Caches and logs need the tenant in their keys as well, or they leak.
- 3
Freshness. Tickets change within minutes and articles within days. Use a change feed for tickets so a new ticket is searchable quickly, and a scheduled sync for articles (Topic 11.2). Show an 'as of' time on answers that come from tickets so staff and customers know how old the facts are.
- 4
Hand-off to a human. Write the rules as code, not as hope: the customer asks for a person; the assistant has had a weak answer twice in a row; the topic is on a never-automate list (refunds above a limit, legal issues, safety); or the tenant's budget is used up. The hand-off carries context: the question, the retrieved sources and a summary, so the human does not start from zero.
- 5
Cost limits. Give each tenant a daily budget in money or tokens, count usage per request, and stop or degrade when it is spent: shorter context, a cheaper model for simple questions, or a hand-off. Alert on a tenant whose spend jumps. The decision function in the example puts the rules in one place so they can be tested.
- 6
Evaluate what matters. Beyond correctness: containment rate (solved without a human), wrong-answer rate on a sample read by staff, hand-off quality, and customer rating. A high containment rate with a rising wrong-answer rate is a failing product, not a succeeding one (Topic 9.8).
Explain it without notes
How do you keep tenants apart in a support assistant?
List the hand-off rules and say why their order matters.
How do you limit cost per customer without breaking the experience?
Why can containment rate mislead, and what pairs with it?
Practice
Write should_hand_off(answered_history) where the list holds True for an answered turn and False for a refusal. Hand off when the last two entries are both False. Print the result as yes or no for three histories.
A dict maps tenants to tokens used today. Print, sorted by name and comma separated, the tenants whose use is at or over a limit of 500.
Trade-offs
- ↔
Shared index with a filter versus an index per tenant: the filter is cheaper to run, a separate index is stronger isolation and simpler to delete.
- ↔
A low evidence floor answers more questions and invents more. A high one hands off more and costs staff time.
- ↔
Cheap model for simple questions: saves money, but needs a router and a measured quality check (Topic 7.6).
Done when you can
Done when you can explain tenant isolation for the index, the caches and the logs.
Done when you can write the hand-off rules in order and defend that order.
Done when you can describe a per-tenant budget and what happens when it is spent.
Done when you can name two metrics that must be read together with containment.