Command Palette

Search for a command to run...

Hectal
Phase 12Advanced13 of 15 in LLM Engineering

Building a Production LLM Service

Put the pieces together: the layers of a real LLM backend, its API, stored conversations, queues, tenants, routing, config, deployment and a service skeleton.

Phases 2 to 11 gave you the parts: a model client, prompts, structured output, streaming, cost controls, tools, retrieval, evals, security and observability. This phase assembles them into one service that other people and programs can call, and asks the questions a single script never faces: what does the request and response look like, where does a conversation live, what happens to a job that takes minutes, and how do two customers share one service without seeing each other's data?

Every example is a small in-memory simulation of one part of the service (a request handler, a conversation store, a job queue on a simulated clock, tenant quotas, a model router, a feature-flag rollout, a health check), so you can run it, read it and break it. Real service code (FastAPI, Spring Boot, a Next.js route handler, a Dockerfile) appears in blocks that are shown, not run, and this site's own lab route is walked through as a production example.

The phase ends with a playbook: a service skeleton in three languages and a pre-launch check. The infrastructure around it is taught elsewhere on Hectal: system design (/system-design) for scaling and queues, Docker (/docker) for images, CI/CD (/cicd) for pipelines and Kubernetes (/kubernetes) for running containers.

0/9 · 0%
9 topics ~3 h 16 code blocks & diagrams
Start with the first topic
1
12.1

Architecture of an LLM Service

A production LLM service is a normal web backend with a few new layers: a gateway that checks who is calling, a prompt layer, a model client, optional retrieval and tools, storage, and observability around all of it. Knowing the layers tells you where each earlier phase plugs in.

21 min 1 code practice

2
12.2

Designing Your Own LLM Endpoint

Your service is an API other programs depend on, so its request, response and error shapes are a contract. Design them small and strict: validate everything, clamp what costs money, return one error shape, support streaming without changing the fields, and version the path so you can change it later.

24 min 3 code practice

3
12.3

Storing Conversations

The model remembers nothing, so your service must store each conversation and send the right part of it every time. That means a database table, a token budget that trims old messages, an optional summary of what was trimmed, strict separation between users, and a way to delete data on request.

21 min 1 code practice

4
12.4

Queues and Background Jobs

Some LLM work takes too long for a web request: summarising a long document, processing a batch, running an agent. Put such work in a queue, return a job id at once, let a worker do it with retries and backoff, and use idempotency keys so repeated submissions do not repeat the work.

22 min 1 code practice

5
12.5

Multi-Tenant Limits and Isolation

When many customers share one LLM service, each needs their own quota, their own data and their own failure domain. The core rules: identify the tenant first, put the tenant into every query, cache key and limit, and test that tenant A can never see tenant B.

19 min 1 code practice

6
12.6

Model Routing

Not every request needs your most expensive model. A router sends easy requests to a cheap, fast model and escalates hard ones to a stronger model, using a confidence or validation signal. Measured on your own examples, it can cut cost a lot, but only if the escalation signal is trustworthy.

21 min 2 code practice

7
12.7

Prompts and Models as Configuration

Prompts, model names, limits and thresholds change much more often than code. Keep them in versioned configuration, switch features with flags, roll changes out to a percentage of users using a stable hash, watch the metrics, and roll back by changing one value instead of redeploying.

20 min 1 code practice

8
12.8

Deploying

Deploying an LLM service means packaging it (usually in a container), giving it its settings and secrets through environment variables, exposing health checks, and respecting the time limits of the platform you run on, which matters because model calls and streams last seconds to minutes.

22 min 3 code practice

9
12.9

Playbook: Service Skeleton

A small service skeleton you can copy: authenticate, enforce a tenant quota, build a budgeted context, call the model, stream events, store the exchange and log the outcome, plus a pre-launch review that checks the production basics.

22 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Clean an incoming chat request

Write clean_request(body) that takes a dictionary and returns a new dictionary with the keys model and max_tokens.

Intermediate
2

Trim a conversation to a budget

messages is a list of (role, text) pairs, oldest first. Write trim(messages, budget) that returns the newest messages whose words add up to at most budget, in the original order.

Intermediate
3

Submit jobs once per idempotency key

Write a class Jobs with submit(key, payload) that returns (job_id, created). The first time a key is seen it creates a job with the id "job-1", "job-2", ... and returns created = True. The same key again returns the same job id and created = False, and creates nothing.

Intermediate
4

Search only the caller's documents

DOCS is a list of (tenant, text). Write search(tenant, word) that returns the list of texts that belong to tenant and contain word as a whole word (compare lower-case words from text.split()), in the original order.

Advanced
5

Route cheap first, escalate when unsure

small(question) and large(question) each return a tuple (answer, confidence). Write route(question, small, large, threshold) that calls small first. If its confidence is at least threshold return (answer, "small").

Advanced
6

Roll out a prompt to a percentage of users

Write bucket(text): start with h = 0, and for each character do h = (h * 31 + ord(ch)) % 1000003; return h * 7919 % 1000003 % 100.

Advanced