PHASE 12Advanced ~22 min· topic 9 of 9

Topic 12.9

Playbook: Service Skeleton

In one line

A small service skeleton you can copy: authenticate, enforce a tenant quota, build a budgeted context, call the model, stream events, store the exchange and log the outcome, plus a pre-launch review that checks the production basics.

Think of it like this

A checklist for opening a shop. Before the first customer you check the lock, the till, the lights, the fire exit and the price tags. None of it is exciting; skipping any one of them is how a good shop has a bad first day. This playbook is that checklist for an LLM service, with code.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Skeleton: A small working starting structure that you fill in.
Request path: The ordered steps one request passes through, from the front door to the stored result.
SSE event: One data: line of a Server-Sent Events stream.
Context: The messages sent to the model for one call.
Pre-launch review: A checklist run before going live.

Step by step

01Step 1: the front door

chat looks the API key up in USERS. An unknown key returns 401 and the single error event, with nothing else touched. The tenant and the user are known from here on.

02Step 2: the quota

The tenant's counter is increased and compared with QUOTA. The third call in the example is refused with 429 and logged, and the model is never called for it.

03Step 3: context and model

context walks the stored history from newest to oldest and keeps what fits the word budget. The new question is added and the fake model streams its reply in pieces. A real client slots in here (see the HTTP block).

Step 3: context and modeldiagram

Rendering diagram…

04Step 4: events and storage

The pieces become delta events, followed by the finish_reason and usage event and [DONE]. Both messages are saved under the tenant and user. The next call's context includes them, which you can see in the log: context=1 on the first call (just the question) and context=2 on the second (one stored reply that fits the 12-word budget, plus the question).

05Step 5: run the review

The second example runs nine checks against two configurations. ready passes all nine, draft fails three and names them. Wire the same idea into CI and the pre-launch conversation becomes a command.

06Step 6: plug in the real model

The HTTP block replaces fake_stream. Keep it the only place that knows the provider URL, key name and response shape, and add the timeout, retry and fallback behaviour there.

Try it yourself

1
Predict the context
Before running, write the context= number for each successful call. Hint: the budget is 12 words, the stored reply "Sure. Reset it from the admin page." has 7 words and the first question has 6, so together they are 13 and only the newest, the reply, fits.
2
Change the quota
Set QUOTA to 3. Which lines change? Then add a second API key for another tenant and show that its first call succeeds after the first tenant is refused.
3
Fail the review on purpose
Set quota to false in the ready configuration. Which line does the review print? Add a tenth check for moderation.

Code & diagrams

The service skeleton: auth, quota, history, events, log New tab

A fake model streams the reply. Each call returns a status and the SSE lines a client would receive.

Expected output

200 9 events, first: data: {"delta":"Sure. "}, last: data: [DONE]
200 9 events, first: data: {"delta":"Sure. "}, last: data: [DONE]
429 1 events, first: data: {"error":"quota"}, last: data: {"error":"quota"}
401 1 events, first: data: {"error":"unauthorized"}, last: data: {"error":"unauthorized"}
tenant=acme status=200 context=1
tenant=acme status=200 context=2
tenant=acme status=429

The pre-launch review as code New tab

Nine production basics checked against a configuration. Run something like this in CI.

Expected output

ready: 9 of 9 pass
draft: 6 of 9 pass
  fix: every query is scoped by tenant
  fix: model calls have a timeout and a fallback
  fix: readiness and liveness endpoints exist

Where the real model plugs inpython

Not run here: needs the network and your own GROQ_API_KEY. Plain HTTP so it never goes out of date; any OpenAI-compatible provider works with a different URL.

# pip install httpx      the model client of the skeleton, using plain HTTP (works with any OpenAI-compatible provider)
import os

import httpx


def call_model(messages, model, max_tokens):
    r = httpx.post(
        "https://api.groq.com/openai/v1/chat/completions",
        headers={"authorization": "Bearer " + os.environ["GROQ_API_KEY"]},  # the key comes from the environment
        json={"model": model, "messages": messages, "max_tokens": max_tokens},
        timeout=30,
    )
    r.raise_for_status()
    body = r.json()
    return body["choices"][0]["message"]["content"], body["usage"]


# In Skeleton.chat, replace fake_stream(messages) with call_model(...), or pass "stream": True and relay the SSE lines.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Save the history under a shared key

In chat, save the history under one fixed key instead of the tenant and user (Python: self.history.setdefault("all", []), and read it with self.history.get("all", []); JavaScript and Java the same with the key "all"). Then add a second API key for another tenant who asks one question.

terminal

$ Run

── what you'll see ──

200 9 events, first: data: {"delta":"Sure. "}, last: data: [DONE]

429 1 events, first: data: {"error":"quota"}, last: data: {"error":"quota"}

200 9 events, first: data: {"delta":"Sure. "}, last: data: [DONE]

401 1 events, first: data: {"error":"unauthorized"}, last: data: {"error":"unauthorized"}

tenant=acme status=200 context=1

tenant=acme status=200 context=2

tenant=acme status=429

tenant=globex status=200 context=2

Myth vs fact

Myth

A skeleton like this is enough for production.

Fact

It is the structure. Real storage, shared quotas, retries, fallbacks, security guards, metrics and deployment still have to be added.

Myth

Streaming and non-streaming need two code paths.

Fact

Build the events once. A non-streaming response is the same events joined into one body.

Myth

The pre-launch review is bureaucracy.

Fact

Each line is a failure someone has had on a real launch. Automating it makes it free.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Treat the request path as a pipeline of small functions with a context object that carries the request id, tenant, user and deadline; each step can be tested alone.
▸
Make the final events carry the numbers you need for billing and debugging: model, usage, finish reason and the request id.
▸
Run the review in CI against a staging configuration, and fail the build on a missing basic; keep a short list of accepted exceptions with owners.
▸
Revisit the checklist after every incident (Topic 11.7): each postmortem usually adds one line.

Remember this

1
Recipe: the request path. When to use it: every endpoint that calls a model. What it guards against: unauthenticated use, runaway cost, cross-tenant leaks, lost history. Order the steps cheapest first: authenticate and find the tenant, check the quota, build the context from stored history within a budget, call the model through the client, stream or return the result, save the exchange, write one log line. The skeleton below does this in about sixty lines and returns SSE events.
2
Recipe: SSE output. When to use it: replies a person reads. What it guards against: long silences and a different shape for streaming and non-streaming. Emit a delta event per piece, one final event with finish_reason and usage, and [DONE] (Topics 5.2 and 12.2). Errors before the stream starts use a normal status and body.
3
Recipe: budgeted history. When to use it: any multi-turn feature. What it guards against: growing cost and overflowing the context window. Keep the newest messages that fit a word or token budget and save both sides of every exchange (Topic 12.3). Adapt it by using the real tokenizer count and adding a summary slot.
4
Recipe: tenant-scoped everything. When to use it: always when more than one customer or user shares the service. What it guards against: data leaks and noisy neighbours. The tenant comes from the key on the server and is part of every counter, history key and log line (Topic 12.5). Adapt it by moving counters to a shared store so all running copies agree.
5
Recipe: the pre-launch review. When to use it: before the first deploy and before each large change. What it guards against: forgetting a basic. The second example turns the checklist into code that returns what still needs fixing, which you can run in CI against your real configuration. Adapt it by adding your own checks, such as eval pass rate or a cost budget alert existing.
6
Recipe: the real model plugs in behind one function. When to use it: to move from the fake to a provider. What it guards against: provider details spreading through your code. The block below shows the model client over plain HTTP in three languages: key from the environment, a timeout, status checking. Add the retries of Topic 6.8 and the fallback of Topic 6.9 inside it, and the rest of the skeleton does not change.
7
What this skeleton leaves out on purpose. Real storage (a database), a shared quota store, retries and fallbacks, a queue for long work (Topic 12.4), routing (Topic 12.6), flags (Topic 12.7), moderation and the security guards of Phase 10, and the metrics and traces of Phase 11. Each plugs into one place in the request path; the earlier topics show how.

Explain it without notes

Walk through the steps of a request in the skeleton and say why they are in that order.

What belongs in the final event of a stream?

What would you add before taking this skeleton to production?

Practice

Write sse(event) that turns a Python dict into one line data: <compact json> (use json.dumps with separators=(",", ":")) followed by a blank line, and print the encoding of {"delta": "Hi "} with repr.

Write join_events(lines) that takes the SSE lines of the skeleton (strings starting with data: ) and returns the text made from all delta events joined together, ignoring other events.

Trade-offs

↔
A single-process skeleton is quick to build and debug; each part you split out gains scale and costs operations.
↔
Streaming improves perceived speed and makes errors, retries and accounting more complex.
↔
A strict pre-launch review slows the first deploy and prevents the incident that would slow you down more.

Done when you can

Before you ship: every request is authenticated and mapped to a tenant on the server.
Before you ship: per-tenant quotas exist and over-quota requests are refused before the model call.
Before you ship: history and caches are keyed by tenant, and a two-tenant test proves it.
Before you ship: max_tokens is clamped, models are allow-listed, and every model call has a timeout and a fallback.
Before you ship: the key comes from the environment, startup fails on missing settings, and liveness and readiness exist.
Before you ship: every request has an id on every log line, with tokens, cost and outcome recorded.
Before you ship: prompts, models and thresholds are configuration with a tested rollback.

Command Palette

Words you'll meet

Step by step

01Step 1: the front door

02Step 2: the quota

03Step 3: context and model

04Step 4: events and storage

05Step 5: run the review

06Step 6: plug in the real model

Try it yourself

Predict the context

Change the quota

Fail the review on purpose

Code & diagrams

Break it on purpose

Save the history under a shared key

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can