PHASE 12Advanced ~24 min· topic 2 of 9

Topic 12.2

Designing Your Own LLM Endpoint

In one line

Your service is an API other programs depend on, so its request, response and error shapes are a contract. Design them small and strict: validate everything, clamp what costs money, return one error shape, support streaming without changing the fields, and version the path so you can change it later.

Think of it like this

The order form at a pizza counter. The form lists what you may choose (size, toppings), refuses an order for a pizza the shop does not make, and always gives you a numbered receipt. If every cashier invented their own form, no customer or kitchen could rely on it. An API contract is that form: it tells callers exactly what to send and what they will get back.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

API contract: The agreed shapes of requests, responses and errors that callers rely on.
Allow-list: A fixed list of accepted values (such as model names). Anything not on it is refused.
Clamp: Force a number into a safe range, for example max_tokens never above your cap.
Error envelope: The one JSON shape every error uses: a type, a message, the field at fault and a request id.
Status code: The number in an HTTP response that says how it went: 200 OK, 400 bad request, 422 invalid, 429 too many requests.
Idempotency key: A caller-chosen unique label on a request; repeating the same key returns the first result instead of doing the work again.
SSE (Server-Sent Events): A way for a server to push a stream of data: lines over one HTTP response (see Topic 5.2).

Step by step

01Step 1: write the contract before the code

Write down one request, one success response and each error as JSON. This is the whole design. The handler below is a function from the raw body text to a status and a body, so it can be tested without any web server at all.

contract.jsonwhole filejson

{"model":"small","max_tokens":256,"messages":[{"role":"user","content":"hi"}]}

{"id":"req_1","model":"small","reply":"...","finish_reason":"stop","usage":{"input":1,"output":4}}

{"error":{"type":"invalid_request","message":"...","param":"model"},"request_id":"req_2"}

02Step 2: parse defensively

The body is untrusted text. Parse it inside a try block and return 400 invalid_json if it fails. Then check it is an object and that messages is a non-empty list. Each check returns the same envelope with the field name in param.

03Step 3: allow-list and clamp

model must be on the allow-list, otherwise 422. max_tokens is clamped with min(value, cap). In the example the last request asks for 9999 and is answered with max_tokens_used: 512, so the caller can see what was applied.

04Step 4: the request id

Every response, success or error, carries the same id the logs carry. When a caller reports a problem they send you req_4 and you find the exact request (Topics 11.2 and 11.3).

05Step 5: put it behind a framework

The framework code in the next block does the same job in FastAPI, Next.js and Spring Boot. Frameworks add schema validation, routing and serialization, but the decisions are yours: allow-list, clamp, one error shape, SSE for streams. The diagram shows the two response paths for one endpoint.

Step 5: put it behind a frameworkdiagram

Rendering diagram…

06Step 6: read a real one

The last block shows the lab route of this very site, shortened and numbered. Items 1 to 9 are the production habits of this topic: defensive parsing, clear 503 for missing config, identify first, allow-listed model, clamped max_tokens, per-user daily limit, key only on the server, an explicit timeout below the platform limit, and an unchanged SSE pass-through.

Try it yourself

1
Add a field
Add an optional temperature field. Accept numbers from 0 to 2 and answer 422 with param: "temperature" otherwise. Which requests in the list should still return 200?
2
Test the errors
Add two requests: a body that is the JSON text [] (valid JSON but not an object) and one whose messages is the text hello. Predict the status and type for each, then run.
3
Compare the frameworks
Look at the real-framework block in the three languages. Find where each one (a) rejects an unknown model, (b) clamps max_tokens, (c) switches to streaming.

Code & diagrams

A chat endpoint as a plain function New tab

No web server: the handler maps a raw body to a status and a JSON body, which is what makes it easy to test.

Expected output

200 {"id":"req_1","model":"small","reply":"You said: hello there","finish_reason":"stop","max_tokens_used":256,"usage":{"input":2,"output":4}}
400 {"error":{"type":"invalid_json","message":"The body is not valid JSON.","param":null},"request_id":"req_2"}
422 {"error":{"type":"invalid_request","message":"messages must be a non-empty list.","param":"messages"},"request_id":"req_3"}
422 {"error":{"type":"invalid_request","message":"Unknown model.","param":"model"},"request_id":"req_4"}
200 {"id":"req_5","model":"large","reply":"You said: explain retries","finish_reason":"stop","max_tokens_used":512,"usage":{"input":2,"output":4}}

The same endpoint in real frameworkspython

Not run here. Python: FastAPI. JavaScript: a Next.js route handler. Java: Spring Boot 3. Each also shows an SSE stream. Two framework defaults differ from the contract above: FastAPI answers invalid JSON and bad fields with 422 (the handler here replaces its body, but you must add your own 400 for broken JSON if you want it), and Spring answers a failed @Valid check with 400 unless you add an exception handler. Check framework versions in your own project.

# pip install fastapi uvicorn      run it with: uvicorn main:app
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, StreamingResponse
from pydantic import BaseModel, Field

app = FastAPI()
ALLOWED_MODELS = {"small", "large"}
MAX_TOKENS_CAP = 512


class Message(BaseModel):
    role: str
    content: str = Field(max_length=6000)


class ChatRequest(BaseModel):
    model: str = "small"
    messages: list[Message] = Field(min_length=1, max_length=16)
    max_tokens: int = Field(default=256, ge=1)
    stream: bool = False


def error(status, kind, message, param=None):
    return JSONResponse(status_code=status, content={"error": {"type": kind, "message": message, "param": param}})


@app.exception_handler(RequestValidationError)  # replace FastAPI's default 422 body with our envelope
async def bad_request(request: Request, exc: RequestValidationError):
    return error(422, "invalid_request", "The request body is not valid.")


async def call_model(req, max_tokens):  # stand-in: your model-client module goes here
    return {"text": "You said: " + req.messages[-1].content, "usage": {"input": 3, "output": 4}}


async def sse_events(req, max_tokens):
    reply = await call_model(req, max_tokens)
    for word in reply["text"].split():
        yield "data: " + word + "\n\n"
    yield "data: [DONE]\n\n"


@app.post("/v1/chat")
async def chat(req: ChatRequest):
    if req.model not in ALLOWED_MODELS:
        return error(422, "invalid_request", "Unknown model.", "model")
    max_tokens = min(req.max_tokens, MAX_TOKENS_CAP)  # clamp, never trust the client
    if req.stream:
        return StreamingResponse(sse_events(req, max_tokens), media_type="text/event-stream")
    reply = await call_model(req, max_tokens)
    return {"reply": reply["text"], "usage": reply["usage"]}

Hectal's own lab route, shortenedtypescript

From src/app/api/llm-lab/route.ts: nine production habits in forty lines.

// src/app/api/llm-lab/route.ts (Hectal's own lab, shortened). Everything here is a production habit.
export const maxDuration = 60;                         // an explicit time limit for the platform

export async function POST(request: Request) {
  let body: Record<string, unknown>;
  try { body = await request.json(); }
  catch { return NextResponse.json({ error: "Bad request." }, { status: 400 }); }   // 1. parse defensively
  if (!aiEnabled()) return NextResponse.json({ error: "..." }, { status: 503 });    // 2. missing config is a clear 503
  const user = await aiUser(request);
  if (!user) return NextResponse.json({ error: "Sign in ..." }, { status: 401 });   // 3. identify the caller first

  const messages = cleanMessages(body.messages);       // 4. keep only known shapes, cap sizes, last 16 messages
  const models = allowed();                            //    allow-listed models only
  const model = typeof body.model === "string" && models.includes(body.model) ? body.model : LAB_DEFAULT_MODEL;
  const sent = { model, messages, max_tokens: num(body.max_tokens, 1, LAB_MAX_TOKENS) ?? 512 };  // 5. clamp max_tokens

  const left = await takeCall(user, { bucket: "lab", limit: labLimit() });
  if (left < 0) return NextResponse.json({ error: "..." }, { status: 429 });        // 6. per-user daily limit

  const { base, key } = providerEndpoint();            // 7. the key exists only on the server
  const res = await fetch(base + "/chat/completions", {
    method: "POST",
    headers: { authorization: "Bearer " + key, "content-type": "application/json" },
    body: JSON.stringify(sent),
    signal: AbortSignal.timeout(55_000),               // 8. a timeout below the platform limit
  });
  if (stream && res.ok && res.body)                    // 9. SSE passes through unchanged
    return new Response(res.body, { headers: { "content-type": "text/event-stream; charset=utf-8", "cache-control": "no-store" } });
  /* ... non-streaming: return the JSON with status, timing and rate-limit headers ... */
}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the client's max_tokens

Remove the clamp: use max_tokens = body.get("max_tokens", 256) (JavaScript: body.max_tokens ?? 256, Java: no Math.min).

terminal

$ Run

── what you'll see ──

200 {"id":"req_1","model":"small","reply":"You said: hello there","finish_reason":"stop","max_tokens_used":256,"usage":{"input":2,"output":4}}

400 {"error":{"type":"invalid_json","message":"The body is not valid JSON.","param":null},"request_id":"req_2"}

422 {"error":{"type":"invalid_request","message":"messages must be a non-empty list.","param":"messages"},"request_id":"req_3"}

422 {"error":{"type":"invalid_request","message":"Unknown model.","param":"model"},"request_id":"req_4"}

200 {"id":"req_5","model":"large","reply":"You said: explain retries","finish_reason":"stop","max_tokens_used":9999,"usage":{"input":2,"output":4}}

Myth vs fact

Myth

Passing the caller's request body straight to the provider is simplest and fine.

Fact

It lets callers choose any model, any size and any hidden option on your key and your bill. Build the provider request yourself from validated fields.

Myth

Error text is only for humans, so its shape does not matter.

Fact

Client code branches on error.type, and an inconsistent shape breaks it. Keep one envelope and stable type codes; messages can change freely.

Myth

Streaming needs a different API.

Fact

It is the same contract with a different delivery: the same fields as events, ending with usage and [DONE].

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Treat the contract as code: keep an OpenAPI description (a machine-readable file that lists every route, field and error of your API; FastAPI generates one, Spring has libraries for it) and run contract tests in CI so a change that breaks a field fails the build (see /cicd).
▸
For long-running generations prefer returning a job id and letting the client poll or subscribe (Topic 12.4); an HTTP request held open for minutes is fragile behind proxies and serverless limits (Topic 12.8).
▸
Return Retry-After on 429 and 503, and put the limits in the docs. Clients that know the rule back off correctly (Topic 6.8).
▸
Decide what is logged for each request: the id, the tenant, sizes and outcome always; the prompt text only with the redaction and retention rules of Topic 11.2.

Remember this

1
Why this exists. The caller of your endpoint may be your own web page today and a partner's server next year. They write code against the shapes you return. If you change a field name, or return an error as sometimes text and sometimes JSON, their code breaks, and the break happens on their side, in production, at night. A written, strict contract is the only protection for both sides.
2
The request. Keep it small: messages (a list of role and content), an optional model from an allow-list you control, an optional max_tokens, and stream. Validate type, size and length of every field, and clamp what costs money: a caller asking for max_tokens: 9999 gets your cap (say 512), not what they asked for. Never pass client fields straight to the provider: a hidden field is a way to reach a model, a tool or a price you did not intend. The /api/llm-lab route of this site does exactly this (see the code below).
3
The response. On success return the reply plus the facts a client needs: an id, the model actually used, finish_reason (Topic 2.3: a client must be able to tell a cut-off reply from a finished one), and usage (input and output tokens). Return the clamped values you actually applied so the caller is not surprised.
4
One error shape. Every failure returns the same JSON envelope: error.type (a stable machine-readable code such as invalid_json), error.message (for humans), error.param (which field), and a request_id. Use the right status: 400 malformed JSON, 401 or 403 who you are, 404 unknown route, 422 well-formed but invalid, 429 limits (with Retry-After), 5xx your service or the provider (Topic 2.6). Do not leak provider error text, stack traces or keys into messages.
5
Streaming and non-streaming share one contract. The same request with stream: true returns Server-Sent Events (Topic 5.2): the same fields, delivered as events, ending with a final event that carries finish_reason and usage, and a closing [DONE]. Decide what an error mid-stream looks like (a final error event) before a client has to guess (Topic 5.7).
6
Versioning. Put the version in the path (/v1/chat). Within a version you may only add: new optional request fields, new response fields. Removing or renaming a field, or changing a meaning, needs /v2. Tell callers that unknown response fields must be ignored. For requests with side effects or long jobs, accept an idempotency key header so a retried request is not done twice (Topic 12.4).

Explain it without notes

What belongs in an LLM endpoint's request, and which fields must you clamp or allow-list?

Why use one error envelope for every failure?

How should streaming and non-streaming responses relate?

How do you change an API without breaking existing callers?

Practice

Add the check temperature (optional, number from 0 to 2) to a small handler validate(body) that returns None when valid or a tuple (422, "temperature") when not. Test it with 0.7, 3 and the text hot.

Write clamp(value, low, high) and use it to turn the requested max_tokens values [10, 600, -5] into applied values between 1 and 512.

Trade-offs

↔
Strict validation rejects some odd but harmless requests; it is easier to loosen a rule later than to tighten one after callers rely on it.
↔
Streaming gives a fast first token but makes errors, retries and proxies harder than a single JSON response.
↔
Clamping silently changes what the caller asked for; returning the applied value makes the change visible.

Done when you can

Done when you can write a request, a success response and an error response as JSON for an LLM endpoint.
Done when you can say which request fields to allow-list and which to clamp.
Done when you can choose the right status for bad JSON, a bad field, a limit and a provider failure.
Done when you can explain how streaming and non-streaming responses share one contract.
Done when you can version an endpoint without breaking callers.

Command Palette

Words you'll meet

Step by step

01Step 1: write the contract before the code

02Step 2: parse defensively

03Step 3: allow-list and clamp

04Step 4: the request id

05Step 5: put it behind a framework

06Step 6: read a real one

Try it yourself

Add a field

Test the errors

Compare the frameworks

Code & diagrams

Break it on purpose

Trust the client's max_tokens

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can