Topic 12.5
Multi-Tenant Limits and Isolation
In one line
When many customers share one LLM service, each needs their own quota, their own data and their own failure domain. The core rules: identify the tenant first, put the tenant into every query, cache key and limit, and test that tenant A can never see tenant B.
Think of it like this
An apartment building. Everyone shares the roof and the water pipes, but each flat has its own lock, its own electricity meter and its own mailbox. If one neighbour runs every tap all day, the others should still get water, and nobody's key should open the next door. A multi-tenant service is the building: one deployment, many tenants (customers), strong walls between them.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Tenant
- One customer of the service, whose data and limits are kept apart from every other customer's.
- Multi-tenant
- One running service shared by many tenants.
- Isolation
- Making sure one tenant cannot read or affect another tenant's data or capacity.
- Quota
- A limit on how much of something a tenant may use in a period.
- Noisy neighbour
- A tenant whose heavy use slows or blocks the others.
- Cache key
- The label under which a cached result is stored; if it leaves out the tenant, tenants share results.
Step by step
01Step 1: the leak
The shared search function looks through every document. Acme asks about a discount and receives Globex's secret discount code. Nothing crashed, no error was logged, and the answer looks helpful. This is why isolation bugs hide.
02Step 2: the fix
search_scoped takes the tenant as a required argument and compares it with each document's owner before looking at the text. Acme now gets "no match", and Globex still finds its own code. The program prints all three lines so you can see the bug and the fix side by side.
def search_scoped(tenant, query):
for owner, text in DOCS:
if owner == tenant and query in text:
return text
return "no match"03Step 3: a quota per tenant
allow counts calls per tenant and returns false after the quota (3). Acme sends five requests in a row: the last two get 429. Globex sends one and is fine. Acme's burst did not touch Globex's allowance.
04Step 4: see where tenant must appear
The picture lists the places a tenant id must appear. If any one of them uses a shared key, that place is where the leak or the noisy neighbour will come from.
05Step 5: a fixed window is the simplest quota
The example counts calls and never resets, as if it were one window. A real quota resets every minute or day, or uses a token bucket (Topic 6.7), and is stored in a shared store such as Redis (a fast database that keeps data in memory) so every running copy sees the same count (see /system-design on rate limiting).
06Step 6: write the test that proves it
The test is the three printed lines turned into assertions: A cannot find B's document, B can, and A's burst does not block B. Add it to CI so a refactor cannot quietly remove the tenant argument.
Try it yourself
- 1
Predict
Before running, write what
search_shared("refund")returns and which tenant's document it is. Then call it for both tenants withsearch_scopedand compare. - 2
Raise one quota
Give Globex a quota of 10 and Acme 3 by making
QUOTAa dictionary. What does the last line print now if Globex sends 6 requests? - 3
Add a cache
Add a dictionary cache for answers keyed by the query only, then by
(tenant, query). Ask Acme and Globex the same question. Which key leaks?
Code & diagrams
Acme must never see Globex's discount code. Prints the bug, the fix and a quota burst.
Expected output
shared: acme asks about 'discount' -> discount code is SAVE20
scoped: acme asks about 'discount' -> no match
scoped: globex asks about 'discount' -> discount code is SAVE20
acme:ok acme:ok acme:ok acme:429 acme:429 globex:okBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the owner check
In search_scoped, remove the owner test so the condition is only query in text (Python: if query in text:; JavaScript and Java the same).
Myth vs fact
Myth
Tenant isolation is the database's job.
Fact
It is every layer's job: the cache, the vector search, the prompt, the logs and the tools can all leak. Check each one.
Myth
The model will not reveal data it was not asked about.
Fact
If the data is in the prompt, the model can quote it. Never put another tenant's data in the prompt in the first place.
Myth
Per-user rate limits protect the provider quota.
Fact
A tenant with many users can still use it all. Limit per tenant as well as per user.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prefer a physical filter that cannot be forgotten: a separate index or schema per tenant for high-value data, or database row-level security that enforces the tenant from the connection, so even a buggy query cannot cross.
- ▸
Attribute tokens and cost to a tenant on every call (Topics 11.4, 12.4). It drives billing, quotas and the conversation about which customer is expensive.
- ▸
Fairness beats raw limits: a weighted fair queue or per-tenant concurrency caps keep latency good for small tenants while a large one runs a batch.
- ▸
Prompt caching and shared system prompts are safe across tenants only if the cached prefix contains no tenant data (Topic 6.6).
Remember this
- 1
Why this exists. Running a separate copy of the service for every customer is expensive, so most services share one. Sharing creates two new risks. First, leaks: customer A's document shows up in customer B's answer. Second, noisy neighbours: one customer's burst of traffic uses the whole provider quota and everyone else gets 429s.
- 2
Identify the tenant first. Every request is tied to a tenant by its API key or login, on the server. The tenant id is never taken from a field the caller can change. From that point the tenant id travels with the request through every layer (Topic 12.1) and is written on every log line.
- 3
Isolate data by construction. The classic LLM leak is retrieval: a search over a shared index returns a document from another tenant, and the model happily quotes it. The same happens with a shared cache (Topics 6.4, 6.5: the cache key must include the tenant), stored conversations (Topic 12.3), uploaded files, tool credentials and the prompt itself. Make the tenant a required argument of every data function, and put the filter in the query (
WHERE tenant_id = ?, or a filter on the vector search, or a separate index per tenant). A check in the caller can be forgotten; a required argument cannot. - 4
Quotas. Give each tenant limits in the units that matter: requests per minute, tokens per day, money per month, concurrent requests, queue length (Topics 6.2, 6.7, 12.4). Reject over-quota requests early, at the gateway, with 429 and
Retry-After. Plans differ, so quotas come from configuration (Topic 12.7), not constants. - 5
Noisy neighbours. One shared provider account has one shared rate limit. If every tenant draws from it freely, one heavy tenant can use it all. Reserve a share per tenant (a token bucket each, Topic 6.7), cap concurrency per tenant, use separate queues or priorities for batch work, and consider separate provider keys for big customers so their usage and limits are theirs alone.
- 6
Keys and secrets per tenant. If tenants bring their own model keys ("bring your own key") store them encrypted, decrypt only at call time, never log them, and never let one tenant's tool call use another tenant's credentials (Topic 10.7).
- 7
Test the walls. Write tests with two tenants that attempt the cross-reads: ask A's question that only B's data can answer, fetch B's conversation id as A, hit the cache with the same question from both. These tests belong in CI (Topic 9.5) and in the red-team suite (Topic 10.8). A tenant isolation bug is a data breach, and it is the kind of bug you do not notice until a customer does.
Explain it without notes
What is a noisy neighbour and how do you contain it in an LLM service?
Where can cross-tenant data leaks happen in an LLM application?
How should the tenant id be established and passed around?
How do you test tenant isolation?
Practice
Write cache_key(tenant, model, question) that returns one string that differs for each of the three inputs and normalises the question to lower case with single spaces. Show that two tenants asking the same question get different keys.
Write TenantLimiter(limit) with allow(tenant) that counts per tenant and refuses after the limit. Show that one tenant's burst does not affect another.
Trade-offs
- ↔
Separate indexes or databases per tenant give the strongest walls and cost more to run and migrate.
- ↔
Shared infrastructure with filters is cheap and efficient, but one missing filter is a breach.
- ↔
Strict quotas protect everyone and can annoy a customer in a legitimate burst; make limits visible and adjustable.
Done when you can
Done when you can explain tenants, quotas and noisy neighbours.
Done when you can list the places where a tenant id must appear.
Done when you can write a data function that cannot be called without a tenant.
Done when you can show how a shared cache key leaks data.
Done when you can describe a two-tenant isolation test.