Topic 12.1
Architecture of an LLM Service
In one line
A production LLM service is a normal web backend with a few new layers: a gateway that checks who is calling, a prompt layer, a model client, optional retrieval and tools, storage, and observability around all of it. Knowing the layers tells you where each earlier phase plugs in.
Think of it like this
A post office. The front desk checks your ID and your stamps (the gateway). The sorting room puts your letter in the right envelope with the right forms (the prompt layer). A courier makes the long, slow trip (the model call). The filing room keeps a copy (storage). And a logbook records every letter that came through (observability). Each room has one job, and when a letter goes missing you know which room to check.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Service
- A program that other programs or people call over a network, running all the time for many callers.
- Gateway
- The front door of the service: authentication, rate limits and basic request checks before any real work.
- Tenant
- One customer (a company or a user account) whose data and limits are kept separate from every other customer's.
- Stateless
- Keeping nothing important in the running copy's memory, so any copy can serve any request and copies can be added or restarted freely.
- Model client
- The one module that talks to the model provider: timeouts, retries, errors, token counts.
- Request id
- A unique label created for each request and written on every log line it causes, so you can follow it through the system.
Step by step
01Step 1: start from the script
The script version is reply = model(question). It works, but it has no caller identity, no limit, no record and no place to put a change. Everything a service adds is a layer around that one line.
02Step 2: draw the layers
The picture shows the path of one request. Solid arrows are the request flow. The dotted arrows show observability: every layer writes to it. Requests that fail at the gateway stop there and are cheap.
03Step 3: build it small
The example below is one class with a handle method. Each layer appends its name to a path list as the request passes through it, so the final log line shows how far each request got: a bad key stops at the gateway, a request over the limit stops at the gateway too, a good one reaches storage.
path = ["gateway"]
if tenant is None:
return finish(401, path)
path.append("prompt")
prompt = TEMPLATE.format(q=question)
path.append("model")
reply = fake_model(prompt)04Step 4: order the checks by cost
Authentication is a dictionary lookup, the rate limit is a counter, the model call costs money and seconds. Put them in that order. Moving the limit check after the model call would still reject the request but would have paid for the model first (see Break it).
05Step 5: swap the fake for the real thing
fake_model has the shape the real client has: text in, reply and token counts out. Replace its body with a call through the provider interface from Topic 2.8 and nothing else in the service changes. That is the payoff of keeping each layer behind a small interface.
06Step 6: know where each earlier phase lives
Phase 2 and Topic 6.8 are the model client. Phases 3 and 4 are the prompt layer and the validation after the model. Phase 5 is how the gateway sends the reply to the caller. Phase 6 is limits and caching in the gateway and client. Phases 7 and 8 are tools and retrieval. Phases 9 to 11 are around the whole thing.
Try it yourself
- 1
Predict the log
Before running, write the five log lines you expect, including the status and the layers. Note which request is the third call from
key-a, and why it stops at the gateway. - 2
Change the limit
Set
LIMIT_PER_MINUTEto 3 and run. Which status changes? Then set it to 1: which two requests are now rejected? - 3
Add a sixth layer
Add a
moderationstep betweenpromptandmodelthat blocks questions containing the wordbanana, with status 400. Where inhandlemust it go so a blocked request never reaches the model?
Code & diagrams
One process, five separate parts. Each log line shows the layers the request reached.
Expected output
status=200 tenant=acme layers=gateway>prompt>model>storage tokens=12+7
status=401 tenant=- layers=gateway tokens=0+0
status=200 tenant=acme layers=gateway>prompt>model>storage tokens=10+5
status=429 tenant=acme layers=gateway tokens=0+0
status=200 tenant=globex layers=gateway>prompt>model>storage tokens=8+3
stored: acme=2 globex=1Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Check the limit after the model call
In handle, move the rate-limit check below the model call, so the order is: look up the tenant, count the call, build the prompt, call the model, then compare with the limit.
Myth vs fact
Myth
A production LLM service needs microservices, a queue and a vector database from day one.
Fact
Most start as one process with clean modules. Add a queue when jobs are slow (Topic 12.4) and a vector store when retrieval needs it (Topic 8.7), not before.
Myth
The model client is just an SDK call.
Fact
It is the place for timeouts, retries, error classes, fallbacks, token counting and usage logging. The SDK call is one line inside it.
Myth
Storing state in a variable is fine if the service is small.
Fact
The first deploy or the second running copy loses or splits it. Keep state in storage from the start; it is only a few more lines.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Create the request id at the edge (or accept a trusted one from the caller or load balancer) and pass it to every layer, every outgoing model call and every log line. It is the single most useful field in an incident (Topic 11.3).
- ▸
A model call takes seconds, so each running copy holds many requests open at once. Size connection pools and worker counts for waiting, not computing: the service is mostly idle while a request is waiting for the provider (see /system-design for capacity thinking).
- ▸
Decide per layer what failure means: a gateway failure returns 4xx quickly, a model failure may fall back (Topic 6.9), a storage failure after a good reply should usually still return the reply and log the loss. Write these decisions down.
Remember this
- 1
Why this exists. Until now you called a model from a script. A service is a program that many callers use at the same time, for days, without you watching. If everything (checking keys, building the prompt, calling the model, saving the result, logging) lives in one long function, you cannot test one concern without the others, you cannot change one without breaking another, and nobody can tell where a slow request spent its time.
- 2
The gateway is the front door. It answers: who is calling (an API key or a login mapped to a tenant, which is one customer or one user's account), are they allowed, have they used too much (Topic 6.2, 6.7 and 12.5), and is the request well formed (Topic 12.2). Requests that fail here never reach the model, so they cost nothing. Do the cheap checks first.
- 3
The prompt layer and the model client. The prompt layer builds the messages from a versioned template plus the user's input, the conversation history and any retrieved documents (Topics 3.7, 3.8, 12.7). The model client is the one place that talks to the provider: timeouts, error classification, retries with backoff, fallbacks, token counting and the key (Topics 2.8, 6.8, 6.9). Nothing else in the service imports an SDK, so changing provider touches one file.
- 4
Retrieval and tools sit between the prompt layer and the model when the feature needs your data (Phase 8) or actions (Phase 7). In a service they have the same rules as anywhere: tool arguments are untrusted input, and a retrieval step must only read the calling tenant's documents (Topic 12.5).
- 5
Storage holds what must outlive one request: conversations (Topic 12.3), job records (Topic 12.4), configuration (Topic 12.7), usage counters. Keep the service itself stateless: no important data in a variable inside one running copy, because a deploy restarts it and a second copy would not see it. With a stateless service you add copies to handle more traffic (see the System Design course at /system-design).
- 6
Observability wraps every layer: a request id created at the gateway and carried through every log line and trace span, metrics for latency, tokens, cost and errors (Phase 11). Without it you cannot answer the first question of every incident: where did the time or the money go?
- 7
Do not over-build. Start as one program with these layers as separate modules or classes. Split a layer into its own service only when there is a real reason: a different scaling need, a different team, or a failure that must be isolated. The simulation below is one process with five clearly separated parts, which is also how many real services begin.
Explain it without notes
Name the layers of an LLM service and say what each is responsible for.
Why should the gateway run before the prompt layer and the model call?
What does it mean for a service to be stateless, and why does it matter for LLM backends?
Why keep the SDK behind a model-client module instead of calling it from request handlers?
Practice
Add a moderation(prompt) layer to the service that returns False for any question containing banana. Make handle return status 400 for blocked questions and log the layers reached. Show the log for two questions, one blocked and one allowed.
Write layer_report(log_lines) that takes the log lines of the example and returns how many requests stopped at each last layer. Print the result for the five example requests.
Trade-offs
- ↔
More layers mean clearer code and easier testing, at the cost of a few more files and indirection.
- ↔
One process is simple to run and debug; splitting layers into services buys independent scaling and isolation but adds network failures and deployment work.
- ↔
Doing cheap checks first saves money but means the gateway needs the data for them (keys, counters) available fast.
Done when you can
Done when you can name the layers of an LLM service and what each does.
Done when you can say which earlier phase each layer comes from.
Done when you can explain why cheap checks come before the model call.
Done when you can explain what stateless means and where state lives instead.
Done when you can sketch one request's path, including where it stops when something is wrong.