Topic 12.8
Deploying
In one line
Deploying an LLM service means packaging it (usually in a container), giving it its settings and secrets through environment variables, exposing health checks, and respecting the time limits of the platform you run on, which matters because model calls and streams last seconds to minutes.
Think of it like this
Opening a food stall at a market. You pack everything in a box that works the same on any table (the container), bring your ingredients and the till key separately and never write the till code on the box (environment variables and secrets), put a sign that says "open" only when the stove is lit (readiness), and know the market closes at a fixed time whatever your queue looks like (platform time limits).
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Container image
- A packaged copy of your program, its libraries and its runtime that runs the same way anywhere.
- Environment variable
- A named setting given to a program by the system that starts it, such as a key or a database address.
- Liveness check
- A question the platform asks: is the process alive? A no means restart it.
- Readiness check
- A question the platform asks: can it take traffic now? A no means stop sending requests.
- Serverless function
- Code run on demand by a platform, which starts and stops copies for you and limits how long each call may run.
- Graceful shutdown
- Finishing the requests in progress before the process exits.
- Probe
- Kubernetes' name for a periodic health check.
Step by step
01Step 1: run it the same everywhere
The Dockerfile below packages the FastAPI service of Topic 12.2. Dependencies come first so the slow install is cached, the app runs as a normal user and the key is not in the image. Build it in CI after tests and evals (see /cicd) and push it to a registry.
02Step 2: fail fast on missing settings
startup_errors returns the names of missing variables. With both present the service starts. Without MODEL_API_KEY it refuses and names the missing one, which is a clear message on deploy day instead of a mysterious 500 at noon.
03Step 3: two health endpoints
liveness always answers 200 as long as the process runs. readiness returns 503 when config or the database is missing, and 200 with a degraded message when only the provider is down. The scenarios show all four cases.
04Step 4: wire the probes
The Kubernetes snippet points the two probes at the two endpoints and reads the key from a Secret. Other platforms have the same two ideas under other names (health check path, startup check).
05Step 5: know your limits
Write the longest legitimate request in your service: a long reply from a slow model, with a retry. Compare it with every timeout between the user and your code: browser, CDN, load balancer, platform function limit, your own client timeouts (Topic 2.7). The shortest one wins, so set your timeouts below it and move longer work to a job.
06Step 6: practise the bad day
Deploy a version that fails readiness and check that traffic never reaches it. Kill the provider key and check that readiness reports degraded and fallback answers still work. Redeploy while a stream is running and check that it finishes. These drills belong in Topic 11.7's runbooks.
Try it yourself
- 1
Break the startup
Remove
DATABASE_URLfrom the second environment too. What does the second start line print now? Why name every missing variable instead of just the first? - 2
Add a startup check
Add a third required value,
ALLOWED_MODELS, and a check that it is not empty. Which of the two environments fails? - 3
Write the Dockerfile for another language
Adapt the Dockerfile idea to a Spring Boot jar: which base image, which files to copy, which command starts it, and which user runs it?
Code & diagrams
Liveness asks only if the process is alive. Readiness reports a provider outage as degraded, not as a failure.
Expected output
start: ok
start: refuse to start, missing MODEL_API_KEY
liveness: 200 alive
readiness all good: 200 ready
readiness provider down: 200 ready (degraded: provider unreachable, fallback answers only)
readiness database down: 503 not ready: database unreachable
readiness booting: 503 not ready: config not loadedA common shape for a FastAPI app. A Next.js or Node service uses the same idea with a Node base image.
# Dockerfile for the FastAPI service of Topic 12.2 (a Python example)
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# Do not run as root, and never bake keys into the image: pass them as environment variables at run time.
RUN useradd --create-home appuser
USER appuser
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
A fragment of a Deployment's container spec. See the Kubernetes course (/kubernetes) for the full object.
# Kubernetes: the two probes call the two health endpoints /healthz and /readyz (your app must serve them).
containers:
- name: llm-service
image: registry.example.com/llm-service:1.4.0
env:
- name: MODEL_API_KEY
valueFrom:
secretKeyRef: { name: llm-secrets, key: model-api-key }
livenessProbe:
httpGet: { path: /healthz, port: 8000 }
periodSeconds: 10
readinessProbe:
httpGet: { path: /readyz, port: 8000 }
periodSeconds: 5
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Make the provider a hard requirement for readiness
In readiness, make the provider check return 503 as well: return 503, "not ready: provider unreachable" (JavaScript: [503, ...], Java: "503 not ready: provider unreachable").
Myth vs fact
Myth
Serverless means no time limits to think about.
Fact
Serverless functions have maximum durations that differ by platform and plan. Long model calls, streams and agents can hit them. Check the current documentation.
Myth
Put the key in the image so it just works.
Fact
Anyone who can pull the image can read the key. Inject secrets at run time and rotate them.
Myth
One health endpoint is enough.
Fact
Restarting a healthy but busy copy because a dependency is down makes things worse. Liveness and readiness answer different questions.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Pin base images by version or digest, scan images for known vulnerabilities in CI, and keep the image small. Both speed and security improve (Docker course, /docker).
- ▸
Use a startup probe or a longer initial delay if the service loads large files or models; a liveness probe that fires before the service has finished starting restarts the copy again and again (a failing readiness probe only stops traffic, it never restarts).
- ▸
Behind a load balancer, set the idle timeout above your longest silent gap in a stream, or send periodic SSE comment lines as keep-alives (Topic 5.4).
- ▸
Propagate cancellation: when a client disconnects, cancel the provider call so you stop paying for tokens nobody reads (Topic 5.5).
Remember this
- 1
Why this exists. Your service works on your laptop with your Python version, your files and your key in your shell. A server has none of that. Deployment is making the service run the same way anywhere, start with the right settings, tell the platform whether it is healthy, and survive being stopped and started often. For LLM services there is one extra pressure: requests are long, so the limits of the platform matter more than for ordinary web apps.
- 2
Containers. A container image packages your code, its dependencies and the runtime, so it runs identically on a laptop, a CI server and production. Write a small
Dockerfile: a slim base image, install dependencies before copying the code (so builds reuse the cache), run as a non-root user, and start one process. See the Docker course (/docker) for images, layers and security, and the CI/CD course (/cicd) for building and pushing the image automatically after the evals pass. - 3
Settings and secrets. Follow the twelve-factor habit (a well-known list of rules for cloud apps): configuration comes from environment variables, not from files baked into the image. The key (
MODEL_API_KEY) and the database URL are injected at run time from a secrets store or the platform's secret feature (Topic 2.4). Check them at startup and refuse to start if one is missing: a service that starts without its key fails later on a real user, and the failure is harder to see. - 4
Health checks. A platform needs to know two different things. Liveness: is the process alive? If not, restart it. Readiness: can it take requests right now? If not, stop sending traffic to it, but do not restart it. Keep liveness trivial. Make readiness check what the service needs to do its job (config loaded, database reachable). Think carefully before making the model provider a hard requirement: during a provider outage every copy would report not ready and you would serve nothing, instead of serving fallback answers (Topic 6.9). Report degraded instead. Kubernetes calls these probes (see /kubernetes).
- 5
Time limits and streaming. Platforms limit how long one request may run, and the number depends on the provider, the plan and the settings, so read your platform's current documentation and do not assume. Serverless functions and many proxies and load balancers have such limits and idle timeouts. A model call, a stream or an agent run can exceed them. You have four options. Stream, because data flowing keeps many connections alive (a total limit may still apply). Set the platform's maximum duration explicitly (this site's lab route exports
maxDuration = 60and times its own provider call out slightly earlier), move long work to a queue with polling (Topic 12.4), or run on long-lived containers instead of functions. Some proxies buffer responses and break streaming until configured otherwise (Topic 5.4). - 6
Graceful shutdown and rollouts. Deploys stop old copies. Give a copy time to finish in-flight requests: stop accepting new ones, wait for current streams to end up to a limit, then exit. Roll out new versions gradually (a few copies first), keep the previous image to roll back to, and run a smoke test on the new copy before it takes traffic (Topic 12.7 for config rollouts).
- 7
Resources and scaling. An LLM service spends most of its time waiting for the provider, so it needs little CPU per request but many concurrent connections. Set memory limits, connection pool sizes and a concurrency limit per copy, scale on concurrent requests or queue age rather than CPU alone, and cap total concurrent provider calls to stay under your rate limit (Topics 6.7 and 12.5). Self-hosted models are a different story (Phase 13).
Explain it without notes
What is the difference between liveness and readiness?
How should an LLM service handle settings and secrets in a container?
Why do platform time limits matter more for LLM services, and what are your options?
How do you deploy a new version without cutting active streams?
Practice
Write readiness(config_loaded, database, provider) returning "ready", "degraded" or "not ready" following the rules of this topic, and print it for the four combinations (True, True, True), (True, True, False), (True, False, True), (False, True, True).
Write missing(env, required) returning the sorted list of required names that are absent or empty in env, and print the result for {"A": "1", "B": ""} with required ["B", "A", "C"].
Trade-offs
- ↔
Serverless scales to zero and has the strictest time limits; long-lived containers cost more when idle and suit long streams and jobs.
- ↔
A strict readiness check avoids sending traffic to a broken copy and can remove all capacity during a shared outage.
- ↔
Small images start fast and have fewer vulnerabilities, but need more care in building.
Done when you can
Done when you can write a Dockerfile that follows the basic good habits.
Done when you can explain environment variables, secrets and fail-fast startup checks.
Done when you can explain liveness against readiness and the degraded state.
Done when you can compare the platform's time limits with your longest request and decide what to do.
Done when you can describe a rolling deployment with graceful shutdown.