PHASE 2Beginner ~23 min· topic 9 of 9

Topic 2.9

Playbook: A Production Model Client

In one line

Two recipes to copy: a model client with a key from the environment, a timeout on every call, error classification and retries with jitter, and a wrapper that writes one safe log line per call with tokens, latency and cost. Both sit behind the same complete interface.

Think of it like this

A company mailroom. Every letter goes through the same desk: it checks the address, uses the right courier, tries again if the courier is busy, refuses to send what cannot be delivered, and writes each item in a logbook without copying the letter's contents. Staff never deal with couriers directly. This playbook is that desk for model calls.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Exponential backoff: Waiting longer after each failed attempt: for example up to 0.5, 1, 2, 4 seconds.
Jitter: Randomness added to a wait so many clients do not all retry at the same moment.
Thundering herd: Many clients retrying together and overwhelming a recovering server.
Structured log: A log line in a fixed format (such as JSON) that tools can search and chart.
Circuit breaker: A guard that stops calling a failing service for a while instead of retrying every request.
Transport: The function that actually sends the HTTP request. Swapping it swaps real for fake.

Step by step

01Step 1: fail early on the key

ModelClient checks MODEL_API_KEY when it is created. A missing key stops the application at start-up with a clear message instead of a confusing 401 on the first request. The last line of the first example shows that error.

02Step 2: one attempt loop

The loop sends the request through the transport with a timeout, treats a timeout like a 408, returns on 200, raises on non-retryable statuses or when attempts run out, and otherwise records a wait and goes round again.

Step 2: one attempt loopdiagram

Rendering diagram…

03Step 3: read the four scenarios

A rate limit (waits 2,000 ms because of Retry-After), then overload (a jittered wait), then success: three attempts. A wrong key: one attempt, not retryable. Four timeouts: three waits, then the error says it was retryable but the client gave up. A cut-off reply: success with finish=length.

04Step 4: wrap it with a logger

The second example logs one JSON line per call: model, outcome, tokens, latency. The failing call is still logged (outcome error, zero tokens). Cost is accumulated from example prices. The final line proves the private prompt text never reached the logs.

05Step 5: plug in the real transport

The last block shows a real transport in each language. It has the same signature as the scripted one, so the client's behaviour is the one you just saw. The real transport turns library timeouts into the client's timeout error.

06Step 6: run the checklist

Go through the checklist below on one of your own calls. Every unticked line is a way to fail in production.

Try it yourself

1
Predict the waits
Before running the first example, predict the number of attempts for each scenario and how many waits are recorded. Which wait comes from a header and which from jitter?
2
Lower the attempt limit
Set max_attempts to 2. What happens to the first scenario, and what does the error say?
3
Log the model's name
In the second example add a feature field to every log line (for example "summary"). Run it in two languages and confirm the lines still match.

Code & diagrams

Recipe: a resilient model client New tab

The transport replays scripted responses. Waits are recorded, not slept, so the example is instant. Jitter uses the course's seeded generator.

Expected output

limit, overload, then ok  -> Hello! finish=stop attempts=3 waits_ms=[2000, 357]
wrong key                 -> ModelError HTTP 401 retryable=no waits_ms=[]
timeouts forever          -> ModelError HTTP 408 retryable=yes waits_ms=[321, 94, 1370]
cut-off reply             -> Hel finish=length attempts=1 waits_ms=[]
no key in the environment -> ModelError MODEL_API_KEY is not set

Recipe: one safe log line per call, with tokens, latency and cost New tab

Prices are example numbers: read yours from configuration. Time is simulated with a counter.

Expected output

{"model":"small-model","outcome":"stop","tokens_in":1200,"tokens_out":40,"latency_ms":820}
{"model":"small-model","outcome":"length","tokens_in":3000,"tokens_out":256,"latency_ms":2400}
{"model":"small-model","outcome":"error","tokens_in":0,"tokens_out":0,"latency_ms":150}
total cost: 0.000538 dollars
prompt text in the logs: no

Where the real transport plugs inpython

Not run here: needs a network and your own key. In JavaScript the client's complete method becomes async because a real transport is.

# Plug a real transport into ModelClient (httpx). Same signature as the scripted one: (body, headers, timeout_ms).
import httpx


def http_transport(body, headers, timeout_ms):
    try:
        r = httpx.post(
            "https://api.groq.com/openai/v1/chat/completions",
            content=body,
            headers={**headers, "Content-Type": "application/json"},
            timeout=timeout_ms / 1000,
        )
    except httpx.TimeoutException as e:
        raise TimeoutError(str(e)) from e  # ModelClient treats it as a 408 and retries
    return r.status_code, dict(r.headers), r.text


# client = ModelClient(http_transport, os.environ)  -> the same retries, classification and errors as in the lab runs above

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Retry everything

In the Python resilient client change is_retryable to return True, then run the first example.

terminal

$ Run

── what you'll see ──

limit, overload, then ok -> Hello! finish=stop attempts=3 waits_ms=[2000, 357]

wrong key -> ModelError HTTP 401 retryable=yes waits_ms=[321, 94, 1370]

timeouts forever -> ModelError HTTP 408 retryable=yes waits_ms=[443, 679, 1146]

cut-off reply -> Hel finish=length attempts=1 waits_ms=[]

no key in the environment -> ModelError MODEL_API_KEY is not set

Myth vs fact

Myth

Retrying more makes the client more reliable.

Fact

Without backoff, jitter and a cap, retries amplify an outage. Few, spaced-out retries of the right errors are what help.

Myth

Logging the prompt and reply is always useful.

Fact

They can contain personal data and secrets. Log metadata by default, and store content only under a retention and redaction policy.

Myth

A wrapper around the client is too much code.

Fact

Each wrapper is about twenty lines, and it replaces the same logging and metering code copied into every feature.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Honour the provider's rate-limit headers (remaining requests and tokens) before you hit 429; the retry path is the last defence, not the first (Topic 6.7).
▸
Add a request id to every log line and pass it to the provider when it accepts one, so a user complaint can be traced to a single call (Topic 11.3).
▸
Decide how errors surface: raising one error type with retryable and status fields lets callers choose a fallback or a user message without parsing strings.

Remember this

1
Recipe: the resilient client. When to use it: for every real model call. It reads the key from the environment and fails at start-up if it is missing (Topic 2.4). It puts a timeout on each call (Topic 2.7) and classifies every status (Topic 2.6). It retries only 408, 429 and 5xx, with a maximum number of attempts. Between attempts it waits for the Retry-After header if present, otherwise uses exponential backoff with full jitter (a random wait between zero and a growing cap), which keeps many clients from retrying at the same moment. What it guards against: crashes on transient failures, endless retries of bugs, hammering a struggling provider. How to adapt it: change the attempt limit and cap from your latency budget, return error types your callers understand, and add a circuit breaker and a fallback model (Topics 6.8 and 6.9).
2
Recipe: the logging and cost wrapper. When to use it: around the client, in every environment. Per call it writes one structured log line (model, outcome, tokens in and out, latency) and adds the cost from your prices. It logs in a finally block so failures are recorded too, and it never writes the prompt or reply text, because those can hold personal data (Topic 10.5). What it guards against: not knowing what a feature costs, silent failure rates, and secrets or private text ending up in logs. How to adapt it: send the line to your logging system as JSON, add a request id and feature name, and sample or redact content only under a policy (Topic 11.2).
3
Recipe: stack them behind the interface. LoggedModel(ModelClient(...)) has the same complete as a fake, so features and tests do not change (Topic 2.8). Put retries inside logging if you want one line per attempt, outside if you want one line per user-visible call. Pick on purpose and write it down.
4
Recipe: always handle the reply, not only the call. The client returns finish_reason, and the caller decides what to do with length (Topic 2.3). The cut-off scenario in the first example shows the client succeeding while the reply is incomplete. Calling code must check it.
5
Checklist: before you ship a model call. Key from the environment and not in code or logs; timeout and total deadline set; only retryable errors retried, with a cap, backoff and jitter; max_tokens set; finish_reason checked; usage logged with cost; the call sits behind an interface and a fake exists for tests; and a plan for the key leaking (rotation steps written down).

Explain it without notes

Walk through what the resilient client does for a 429 with Retry-After, a 401 and a timeout.

Why does the logging wrapper write its line in a finally block and leave out the prompt text?

Why add full jitter to exponential backoff?

Practice

Write backoff_cap(attempt, base_ms=500, max_ms=8000) that returns min(max_ms, base_ms * 2**attempt) and print it for attempts 0 to 5.

Write log_line(model, outcome, tokens_in, tokens_out, latency_ms) that returns the compact JSON text of a dictionary with those five fields in that order.

Trade-offs

↔
More retry attempts raise success rates but add latency and load; set the cap from your latency budget.
↔
Rich logs help debugging but raise privacy and cost; metadata-first with opt-in content is the safe default.

Done when you can

Before you ship: the key comes from the environment and a missing key stops start-up.
Before you ship: every call has a timeout and a total deadline, and max_tokens is set.
Before you ship: only 408, 429 and 5xx are retried, with a cap, backoff and jitter, and Retry-After is honoured.
Before you ship: finish_reason is checked on every reply.
Before you ship: every call logs model, tokens, latency, cost and outcome, without prompt text.
Before you ship: the call sits behind an interface and a fake exists for tests.

Command Palette

Words you'll meet

Step by step

01Step 1: fail early on the key

02Step 2: one attempt loop

03Step 3: read the four scenarios

04Step 4: wrap it with a logger

05Step 5: plug in the real transport

06Step 6: run the checklist

Try it yourself

Predict the waits

Lower the attempt limit

Log the model's name

Code & diagrams

Break it on purpose

Retry everything

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can