Topic 0.4
Why "Just Call the API" Fails in Production
In one line
A demo calls the model once and works. A product makes thousands of calls and meets rate limits, timeouts, broken formats, wrong answers, attacks and model updates. Each failure has a known engineering fix, and this course teaches them in order.
Think of it like this
Cooking one meal for a friend vs running a restaurant. At home, if the gas runs out you wait; if you burn the rice you make more. A restaurant serving 500 people needs backup gas, a recipe card so every plate tastes the same, someone checking each plate before it leaves, and a plan for the night the supplier doesn't show up. Same cooking, very different engineering.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Production
- The real system that real users use, as opposed to a demo or a test.
- Rate limit
- A cap on how many requests or tokens you may send per minute. Going over returns HTTP 429.
- Timeout
- A time limit after which you stop waiting for an answer.
- Retry
- Trying a failed call again, for errors that may succeed the next time.
- HTTP status code
- A number in every web response that says how it went: 200 OK, 429 too many requests, 503 overloaded.
- Prompt injection
- Text that tries to make a model ignore its instructions (Phase 10).
Step by step
01Step 1: the demo works
The demo classifies a review: one call, json.loads, read the field. On the author's laptop, the first request returns clean JSON and everyone is impressed.
02Step 2: replay real traffic
The first example replays five responses of the kinds a real API returns over a day: clean JSON, a 429 rate limit, JSON wrapped in friendly chatter, a timeout, and clean JSON again. A small harness runs the demo on each and records how it ended. Three of five crash.
03Step 3: add three small guards
One retry for errors worth retrying (rate limit, timeout). Cut the JSON object out of any extra text before parsing. If there is still no usable answer, return a safe fallback (unknown) instead of crashing. The second example runs the same kinds of trouble, plus a reply with no JSON at all: nothing crashes, and one request honestly answers unknown.
04Step 4: know what the quick guards don't fix
Retrying immediately makes rate limits worse when many clients do it together (Topic 6.8 adds backoff and jitter). Cutting out {...} breaks when the text contains two objects or braces in strings (Phase 4 uses JSON mode and validation). A fallback hides failures unless you log and count them (Phase 11). Quick guards are a start, not the finish.
05Step 5: the map of the course
Every production failure has a home: errors and timeouts in Phase 2, prompts in Phase 3, format in Phase 4, waiting in Phase 5, cost and limits in Phase 6, missing abilities in Phase 7, missing knowledge in Phase 8, silent regressions in Phase 9, attacks in Phase 10, blindness in Phase 11, scale in Phase 12, and the model choice itself in Phase 13.
Try it yourself
- 1
Predict before running
Before running the first example, write down which requests you expect to crash and why. Then run and compare.
- 2
Two failures in a row
In the second example, change request 2's script so both attempts are
rate_limited. Run it. What does the request return, and what note explains why? - 3
Break the JSON cutter
Give request 3 the reply
Here: {"sentiment": "positive"} or maybe {"sentiment": "negative"}. What doesparse_labeldo with two objects, and why is "find the first{and last}" fragile?
Code & diagrams
A scripted fake API replays five kinds of real responses. The demo code has no guards.
Expected output
request 1: positive
request 2: CRASH (rate limited, HTTP 429)
request 3: CRASH (reply was not valid JSON)
request 4: CRASH (timed out)
request 5: neutral
3 of 5 requests crashed the demoOne retry, cutting JSON out of extra text, and a safe fallback. Phases 4 and 6 build the production versions.
Expected output
request 1: positive
request 2: negative (retried after rate limit)
request 3: negative (cut JSON out of extra text)
request 4: positive (retried after timeout)
request 5: unknown (no JSON, used fallback)
0 of 5 requests crashed; 1 answered with the fallbackBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Retry everything
In the second example's guarded_classify, catch every exception in the retry loop instead of only rate limits and timeouts (in Python except Exception as e, in Java catch (RuntimeException e), in JavaScript remove the instanceof check). Then make fake_api raise a ValueError("bad request") for request 1.
Myth vs fact
Myth
If it worked in testing, it works in production.
Fact
A few manual tries can't show failures that happen 1 time in 100. Replay real traffic and run evaluation sets (Phase 9).
Myth
The SDK handles errors for me.
Fact
SDKs retry some errors by default, but they don't fix wrong formats, wrong answers, costs or injection, and their defaults may not suit your latency budget.
Myth
A crash is the worst failure.
Fact
A quiet wrong answer is often worse: nothing alerts you, and users trust it. Validation and evaluation catch what crashes don't.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Count failures by type in production: 429s, timeouts, parse failures, fallbacks,
finish_reason: "length". Each counter maps to one fix; without them you're guessing (Topic 11.4). - ▸
Fallbacks need product design: "unknown" must be a state the UI and downstream systems handle (a human queue, a neutral label), not a string that breaks the next step.
- ▸
Retries multiply load during an outage. Cap them, add jitter, respect
Retry-After, and use a circuit breaker so a provider outage doesn't become your outage (Topics 6.8, 6.9).
Remember this
- 1
Why demos lie. A demo is one happy request, made by its author, with a prompt they tuned until it worked. Production is many requests, from people who type anything, at the same time, every day, against a service you don't control. Problems that happen 1 time in 100 never show up in a demo and happen hundreds of times a day in a product.
- 2
It fails loudly. The API returns errors: rate limits (HTTP 429: too many requests or tokens per minute), timeouts (no answer in time), overload (503), bad requests (400: something you sent is wrong). Code that assumes success crashes. Fixes: classify errors, retry the retryable ones with backoff and jitter, set timeouts, fall back to another model (Topics 2.6, 2.7, 6.8, 6.9).
- 3
It fails quietly. The call succeeds, but the content is wrong for your program: JSON wrapped in chatter ("Sure! Here is the JSON:"), a missing field, a cut-off reply (
finish_reason: "length"), a confident wrong fact, or a different answer than yesterday. Nothing crashes, so nothing alerts you. Fixes: structured output and validation (Phase 4), grounding with your data (Phase 8), evaluations (Phase 9). - 4
It fails expensively and slowly. Every call costs money per token, and long prompts or chatty answers multiply it. A feature that costs 0.1 cents per call costs 1,000 dollars at a million calls. Users also notice a two-second wait. Fixes: token budgets, caching, smaller models, streaming (Phases 5 and 6).
- 5
It fails under attack and over time. Users try to make the bot ignore its rules (prompt injection), leak data or run harmful actions (Phase 10). Providers update or retire models, and your carefully tuned prompt starts behaving differently (Topic 3.8, Phase 9). And when something goes wrong you need logs, traces and metrics to see it (Phase 11).
- 6
The good news. Each failure has a known fix, most of them small. The examples below replay five real kinds of response. The demo code crashes on three of five. The same code with three small guards (one retry, cutting JSON out of extra text, a safe fallback) handles all five. The rest of the course turns those quick guards into production-grade versions.
Explain it without notes
List five ways a model call can fail in production, split into loud and quiet failures.
Why can a feature pass every manual test and still fail daily in production?
Which errors should you retry, and which shouldn't you?
Practice
Write is_retryable(status) that returns True for 408, 429 and any 5xx status, and False otherwise.
Write extract_json_object(text) that returns the parsed object from the first { to the last }, or None if that slice doesn't parse.
Trade-offs
- ↔
Retries raise success rates but add latency and load; cap them.
- ↔
Fallbacks keep the product running but can hide problems; always log and count them.
- ↔
Each guard adds a little code; skipping them moves the cost to users and on-call engineers.
Done when you can
Done when you can list loud and quiet LLM failures and an example of each.
Done when you can explain which errors are worth retrying.
Done when you can add a retry, a JSON extractor and a fallback to a naive call.
Done when you can name which phase of the course fixes each failure.