PHASE 6Intermediate ~26 min· topic 11 of 11

Topic 6.11

Playbook: Resilient, Affordable Calls

In one line

The phase's recipes in one place: a client that wraps any model with a cache, a rate limiter, retries with jitter, a fallback chain and a cost guard, and a per-feature report that turns your call log into costs, p95 latency and alerts.

Think of it like this

A well-run delivery kitchen. Orders already cooked today are handed out from the warm shelf (cache). A queue manager keeps the number of orders on the stove under control (limiter). If a burner fails, the cook retries once, then moves to the second stove (retry and fallback). The owner has a daily food budget and a board showing orders, delays and waste for each dish (cost guard and report). None of it is the cooking, and without it the kitchen collapses on a busy Friday.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Resilient client: A wrapper around a model call that survives rate limits, failures and outages and keeps cost under control.
Wrapper (decorator): Code that has the same interface as the thing it wraps and adds behaviour around it.
Worst-case reserve: Counting the most a call could cost against the budget before making it.
Per-feature report: Calls, errors, latency percentiles and cost grouped by feature.
Alert threshold: The level of a metric (such as p95 above 2 seconds) at which people are told.
Configuration: Values you can change without changing code: prices, limits, model lists, switches.

Step by step

01Step 1: the layers, in order

Cache, then budget check, then limiter, then call, then retry and fallback on failure, then meter and fill the cache on success. The order is not arbitrary: each layer protects the ones after it from work they would waste.

Step 1: the layers, in orderdiagram

Rendering diagram…

02Step 2: the fake models tell a story

The primary model answers 429 on its first call, works on the second, and then fails with 503 for ever. The backup always works. Five requests then exercise every layer: a retry that succeeds, a cache hit, a fallback, a second fallback and a refusal by the budget.

03Step 3: read the output as a log

Request 1 comes from the primary after one jittered retry. Request 2 is the same question with extra spaces and comes from the cache with no call. Requests 3 and 4 fail twice on the primary and fall back. Request 5 is refused before any call because the worst case no longer fits the budget. The last line adds up the spend and the calls per model.

04Step 4: plug the real model in

The real-SDK block shows an OpenAI-compatible model class with the same complete method. It maps 429 to RateLimited, timeouts and 5xx to ServerBusy, and lets 400-class errors propagate. Nothing else changes.

05Step 5: add the report

The second example reads a call log and prints one line per feature plus alerts. The summary feature has a 6-second timeout that pushes its p95 over the goal, and the search feature has a 429 and one very slow call. The log lines come from the same wrapper: record start time, end time, tokens and outcome around every call.

06Step 6: the before-you-ship list

The checklist at the end of the topic is the phase in one page. Walk through it before every launch of an LLM feature.

Try it yourself

1
Predict the run
Before running the client example, write down which requests come from the cache, the primary and the backup, and which is refused. Then run it and compare.
2
Raise the budget
Set BUDGET to 700. Which request is refused now, if any? Then set it to 200 and find the first refused request.
3
Make the primary fail longer
Change the primary's script to ["503"]. How many times is the primary called now, and how does the time t of each request change?
4
Tighten the alert
In the report, set P95_ALERT_MS to 1000. Which features alert now? Does the chat feature, which looks healthy, stay quiet?

Code & diagrams

Recipe: the resilient client New tab

Cache, budget guard, limiter, retry with full jitter, fallback chain and cost meter around a fake model. Prices, limits and times are example numbers on a simulated clock.

Expected output

request 1: from primary at t=100
request 2: from cache at t=1000
request 3: from backup at t=2268
request 4: from backup at t=3200
request 5: refused, the budget is used up (spent 420)
spent 420 of 500 micro-dollars, primary called 6 times, backup 2

Recipe: a per-feature cost and latency report New tab

Twelve made-up log lines. Prices and alert thresholds are example numbers: set your own in configuration.

Expected output

chat     calls 4  errors 0  p50 640 ms  p95 700 ms  cost 3280 micro-dollars
search   calls 4  errors 1  p50 380 ms  p95 2900 ms  cost 3170 micro-dollars
  ALERT search: p95 above 2000 ms
  ALERT search: 1 of 4 calls failed
summary  calls 4  errors 1  p50 1600 ms  p95 6100 ms  cost 10880 micro-dollars
  ALERT summary: p95 above 2000 ms
  ALERT summary: 1 of 4 calls failed

Where the real model plugs inpython

Not run here: needs a network and your own GROQ_API_KEY. It raises the RateLimited and ServerBusy errors of the resilient client. SDK retries are switched off so retries are counted in one place.

# pip install openai      (Groq and many other providers speak the OpenAI-compatible API)
import os

import openai
from openai import OpenAI

# max_retries=0: the ResilientClient does the retrying, so the SDK must not stack its own retries underneath.
client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.environ["GROQ_API_KEY"], timeout=20.0, max_retries=0)


class OpenAICompatibleModel:
    """Plugs in where FakeModel was: the same complete() method, and the same two retryable exceptions."""

    def __init__(self, name):
        self.name = name  # for example "openai/gpt-oss-20b"

    def complete(self, prompt):
        try:
            reply = client.chat.completions.create(model=self.name, messages=[{"role": "user", "content": prompt}], max_tokens=300)
        except openai.RateLimitError as e:  # HTTP 429
            raise RateLimited() from e
        except (openai.APITimeoutError, openai.APIConnectionError) as e:  # no reply at all
            raise ServerBusy() from e
        except openai.APIStatusError as e:
            if e.status_code >= 500:
                raise ServerBusy() from e
            raise  # 400, 401, 404 ... are our own bug: do not retry them
        return reply.choices[0].message.content or ""

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Check the budget after the call

In ask, move the if self.spent + CALL_COST > BUDGET: raise BudgetExceeded() check to just after self.spent += CALL_COST and change it to if self.spent > BUDGET: (in every language), so the cost is counted first and checked afterwards.

terminal

$ Run

── what you'll see ──

request 1: from primary at t=100

request 2: from cache at t=1000

request 3: from backup at t=2268

request 4: from backup at t=3200

request 5: refused, the budget is used up (spent 560)

spent 560 of 500 micro-dollars, primary called 8 times, backup 3

Myth vs fact

Myth

A resilient client means calls never fail.

Fact

It means failures are survived, counted and bounded. Some requests still end in a canned reply or a refusal, and the product must handle them.

Myth

One layer is enough: a retry library covers it.

Fact

A retry doesn't limit rate, cap spend, or move to another model. Each layer covers a different failure.

Myth

The SDK's default retries are fine, so stack mine on top.

Fact

Two retry layers multiply attempts (3 × 3 = 9). Choose one layer and switch the other off.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Put the wrapper behind the same interface as the raw client and compose small pieces: Cached(Budgeted(Retrying(Limited(RealModel)))), with the cache outermost so a hit skips everything inside. Each piece is testable alone, and you can reorder or remove one.
▸
Emit a structured log line for every attempt, not just every request: model, attempt number, status, wait, tokens, cost, cache hit. Retries and fallbacks are invisible without it (Phase 11).
▸
Run the failure paths regularly: a game day where the fake or a proxy returns 429s and 503s in staging. Resilience code that has never run is a guess.
▸
Keep every number from this phase (limits, timeouts, TTLs, prices, thresholds) in one reviewed configuration file with owners and comments, and revisit it when the provider or pricing changes.

Remember this

1
Recipe: the resilient client. *When to use it:* every feature that calls a model from a server. *What it guards against:* repeated calls costing money, rate limits, transient failures, a provider outage, and a runaway bill. *How to adapt it:* it is a wrapper around the same complete(prompt) interface the fake model has (Topic 0.7), so the feature code never changes. The order matters. Check the cache first (the cheapest call is the one you don't make). Then, for each model in the fallback chain (Topic 6.9), loop over attempts. Before each attempt, check the budget using the worst-case cost (Topic 6.2) and wait for the limiter (Topic 6.7). Then call. On a retryable failure, wait with full jitter and try again (Topic 6.8). On success, meter the cost and fill the cache.
2
Recipe: classify errors once. *When to use it:* always, at the edge where the SDK or HTTP client meets your code. *What it guards against:* retrying bugs, and missing real outages. *How to adapt it:* translate the client's errors into two of your own, RateLimited (429) and ServerBusy (5xx, timeouts, dropped connections). Everything else (400, 401, 404) propagates untouched so it fails loudly. The real-SDK block below shows the translation for the OpenAI-compatible client in three languages, and turns off the SDK's own retries (max_retries=0) so retries are counted in one place.
3
Recipe: the limiter and the retry share one clock. *When to use it:* whenever you run more than one call at a time (Topic 6.10). *What it guards against:* retries that ignore the rate limit and make 429s worse. *How to adapt it:* in the example both the limiter's turn and the jittered waits move the same now. In production they share a token bucket (Topic 6.7) across all workers, and a 429's Retry-After is the minimum wait.
4
Recipe: reserve the worst case, then settle. *When to use it:* any per-user or global budget (Topic 6.2). *What it guards against:* a budget crossed by the call that checks it. *How to adapt it:* check spent + worst case before the call (the example uses a fixed cost per call; real code estimates input tokens and adds max_tokens), and after the call replace the estimate with the real usage. In the example, request 4 is the last call that fits, and request 5 is refused before any model is called.
5
Recipe: the per-feature report. *When to use it:* from the first day in production. *What it guards against:* finding out from the invoice, or from angry users, that one feature has become slow, failing or expensive. *How to adapt it:* log one record per call (feature, latency, tokens, outcome). Aggregate by feature: calls, errors, p50 and p95, cost. Alert on p95 above a goal and on an error share above a limit. The second example turns twelve log lines into three report lines and two alerts.
6
Recipe: what to put in configuration. *When to use it:* before you ship. *What it guards against:* code changes during an incident. *How to adapt it:* prices, model chain, timeouts, attempt counts, backoff base and cap, rate limits, cache TTL, budgets, alert thresholds and the kill switch (Topic 6.2) all live in configuration the on-call person can change. The code never contains a price or a model name that you can't change without a deploy.
7
Recipe: test the bad day. *When to use it:* in CI. *What it guards against:* resilience code that has never run. *How to adapt it:* fake models make failure free and repeatable: scripted 429s, 503s, timeouts and slow replies, on a simulated clock. The first example's scripted run is exactly such a test, and the exercises of this phase build the pieces so you can write more.

Explain it without notes

Walk through the order of the layers in the resilient client and justify it.

Why translate errors into RateLimited and ServerBusy at the edge?

What does the per-feature report tell you that the provider's bill does not?

List the values that belong in configuration rather than code, and say why.

Practice

Write ask_with_fallback(models, prompt) where models is a list of functions that take a prompt and either return text or raise RuntimeError. Return (index, text) for the first one that works, or (-1, "canned") if all fail. Test with a first function that fails and a second that works.

Write report_line(feature, latencies, costs) returning "<feature> calls N p95 X ms cost Y" where p95 uses the nearest-rank method and cost is the sum of costs. Print it for "chat", [100, 200, 300, 400] and [5, 5, 5, 5].

Trade-offs

↔
Every layer adds code and a way to fail; add the ones whose failure you have actually seen or can price.
↔
Aggressive caching and cheap fallbacks cut cost and may cut quality; measure with an evaluation set (Phase 9).
↔
A strict budget protects money and refuses real users sometimes; pair it with alerts so you raise it in time.

Done when you can

Before you ship: every call has a timeout, max_tokens, and a bounded input length.
Before you ship: errors are classified once, and only 408, 429, 5xx and timeouts are retried, with full jitter and Retry-After respected.
Before you ship: there is one retry layer, with attempts and a total deadline capped, and SDK retries are switched off or accounted for.
Before you ship: a rate limiter and a concurrency limit keep you under the provider's requests and tokens per minute.
Before you ship: there is a fallback chain (or a reasoned decision to have none) and a designed degraded reply, tested with fake failures.
Before you ship: a per-user limit, a global ceiling with an alert, and a kill switch exist, and the kill switch has been flipped in staging.
Before you ship: the cache key includes everything that changes the answer, and nothing personal or random is cached.
Before you ship: every call is logged with feature, model, latency, tokens, cost and outcome, and a report alerts on p95 and error share.

Command Palette

Words you'll meet

Step by step

01Step 1: the layers, in order

02Step 2: the fake models tell a story

03Step 3: read the output as a log

04Step 4: plug the real model in

05Step 5: add the report

06Step 6: the before-you-ship list

Try it yourself

Predict the run

Raise the budget

Make the primary fail longer

Tighten the alert

Code & diagrams

Break it on purpose

Check the budget after the call

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can