Topic 6.9
Fallbacks, Deadlines and Circuit Breakers
In one line
When your main model is slow or down, a fallback chain tries the next model within a deadline, a circuit breaker stops you calling a provider that keeps failing, and graceful degradation gives users something useful instead of an error.
Think of it like this
A restaurant whose main oven breaks. The cook tries the second oven, then the grill, and if dinner would be later than the guests will wait, the waiter offers cold plates now. Meanwhile, the manager puts a sign on the broken oven ("do not use for an hour") so nobody wastes time walking to it, and checks it again later with one small tray.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Fallback
- A second choice used when the first fails.
- Fallback chain
- An ordered list of models to try, best first.
- Deadline
- The latest time by which the whole operation must finish.
- Per-attempt timeout
- The longest one try may take before you give up on it.
- Graceful degradation
- Giving a simpler or partial result instead of failing completely.
- Circuit breaker
- A switch that stops calling a failing service for a while.
- Closed / open / half-open
- Breaker states: calls flow / calls are skipped / one trial call is allowed.
Step by step
01Step 1: one model, one point of failure
Your feature calls one model. When that model has a bad hour, your feature has a bad hour. Retries only help if the failure is short.
02Step 2: an ordered chain
Write the models as a list. Try each in order, stopping at the first success. Only fall through on errors that justify it (the retryable ones from Topic 6.8: 429, 5xx, timeout), not on a 400 that every model would reject.
CHAIN = ["big-model", "medium-model", "small-model"]
for model in CHAIN:
result = call(model)
if result == "ok":
break03Step 3: add a deadline
Track the time spent. Before each attempt, compute the time left; stop if none is left, and give each attempt a timeout of at most the time left. The output of the first example shows three requests: request 1 falls to the second model after a quick 429, request 2 falls through two models, and request 3 runs out of time.
04Step 4: decide what "degraded" means
Request 3 gets the canned message. A better product might return a cached answer for a similar question (Topics 6.4, 6.5) or a shorter answer from a very fast model. The right choice depends on the feature; the code path must exist and be tested.
05Step 5: skip a provider that is clearly down
A breaker counts failures. After three in a row it opens, and for the next five seconds calls are skipped: no waiting for timeouts, no extra load. In the output, t=5000 to t=8000 make no call at all.
06Step 6: probe carefully
After the cool-down the breaker lets one trial call through. At t=9000 the provider is still down, so the trial fails and the breaker opens again. At t=14000 the next trial works and normal service resumes. One probe, not a flood, is what lets a struggling provider recover.
07Step 7: watch it
Count breaker opens, fallback uses and degraded replies, and alert on them (Topic 11.4). A fallback that works silently for a month means you pay for a model you aren't really using, or that quality has changed without anyone noticing.
Try it yourself
- 1
Predict request 3
Change
DEADLINE_MSto 3500 and predict request 3 before you run it. Does the small model get a turn? How long does it take? - 2
Order matters
Swap the chain so the small model is first. Which requests answer faster? What do you give up?
- 3
Tune the breaker
Set
OPEN_MSto 2000. At which times is a trial call made now, and how many calls are wasted on the down provider before it recovers?
Code & diagrams
Each model's behaviour is scripted. The deadline and timeouts are on a simulated clock.
Expected output
request 1: medium-model after 750 ms
path: big-model 429 (50 ms) -> medium-model ok (700 ms)
reply: from medium-model
request 2: small-model after 2000 ms
path: big-model timeout (1500 ms) -> medium-model 503 (100 ms) -> small-model ok (400 ms)
reply: from small-model
request 3: canned message after 3000 ms
path: big-model timeout (1500 ms) -> medium-model timeout (1500 ms) -> deadline reached
reply: Sorry, I can't answer right now. Please try again in a minute.The provider is down from t=2000 to t=10000. One call is attempted per simulated second.
Expected output
t=0: ok, state closed
t=1000: ok, state closed
t=2000: failed, state closed
t=3000: failed, state closed
t=4000: failed, state open
t=5000: skipped (no call made), state open
t=6000: skipped (no call made), state open
t=7000: skipped (no call made), state open
t=8000: skipped (no call made), state open
t=9000: failed, state open
t=10000: skipped (no call made), state open
t=11000: skipped (no call made), state open
t=12000: skipped (no call made), state open
t=13000: skipped (no call made), state open
t=14000: ok, state closed
t=15000: ok, state closed
t=16000: ok, state closedBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the half-open step
In allow, remove the lines that set the state to half-open, so an open breaker simply stays open (in every language). Run it.
Myth vs fact
Myth
A fallback model gives the same quality.
Fact
Usually it doesn't. Test your prompts and checks on every model in the chain and log which one answered.
Myth
More retries make a fallback unnecessary.
Fact
Retries wait for a recovery. Fallbacks and breakers keep serving users while the provider is down.
Myth
A circuit breaker hides outages.
Fact
It limits how much damage an outage does to you. You still alert on it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Fall back across providers, not only models: a model on the same provider shares the provider's outage. Keep keys, clients and prompts ready for the second provider.
- ▸
Hedged requests: for slow-but-important calls, start a second call to another model after a short delay and take the first reply. It trades extra cost for a shorter tail (Topic 6.3), so use it sparingly.
- ▸
Make breaker state per provider and per model, and share it across servers if possible. Otherwise each server learns the outage separately and sends its own probes.
- ▸
Count only the failures that say the provider is unhealthy (timeouts, 5xx, and usually 429) towards opening the breaker. A 400 or 401 is your own bug and would open it for no reason. A failure-rate rule over a window (for example, more than half of the last 20 calls) is more robust than "N in a row" when traffic is mixed, but needs a minimum number of calls so one early error cannot open it.
- ▸
Test failure on purpose: a fake model that fails in staging proves the chain, the deadline and the degraded UI work before a real outage.
Remember this
- 1
Why this exists. Retries (Topic 6.8) handle a short hiccup. They don't help when one model or one provider is down or slow for minutes. If your product depends on exactly one model, your uptime is the provider's uptime, minus your own bugs. A fallback chain turns "the provider is down" into "answers are a bit less good for a while".
- 2
The fallback chain. An ordered list of models, best first: try the first; on a retryable failure (429, 5xx, timeout), try the next; and so on. Later entries are usually smaller, faster, cheaper, or from another provider. This site's AI tutor (
src/lib/ai/provider.ts) does exactly this: when a model is rate-limited (429) or fails with a server error, it tries the next model in its list, and the learner just gets an answer. Because model names change often, the list can be overridden in configuration. - 3
Deadlines. A chain without a time limit can take longer than any user will wait: three timeouts in a row is a minute. Set a deadline for the whole operation, and a per-attempt timeout that never exceeds the time left. The first example uses a 3-second deadline: request 3 spends 1.5 s on each of the first two models and then has no time left, so it stops and degrades instead of waiting for the third.
- 4
Degrade gracefully. When every option fails or time runs out, show something honest and useful: a cached answer, a shorter or simpler answer, a "try again in a minute" message, or a queue ("we'll email you"). The worst outcomes are a spinner that never ends or a raw stack trace. Decide the degraded experience on purpose, test it, and make the UI able to show it.
- 5
Fallback quality differs. A smaller model may follow your format worse, refuse differently or know less. Test your prompts and output checks against every model in the chain (Phase 9), and validate the output of every fallback (Phase 4). Log which model answered each request, so you notice when most traffic quietly runs on the fallback.
- 6
The circuit breaker. If a provider fails ten times in a row, trying it again costs a timeout each time and adds load to a struggling service. A circuit breaker watches failures. Closed is normal: calls go through. After a number of failures in a row it opens: calls are skipped at once (the fallback is used immediately) for a cool-down period. Then it goes half-open: one trial call is allowed. If it works, the breaker closes; if it fails, it opens again. The second example prints every step against a provider that is down from t=2000 to t=10000.
Explain it without notes
How does a fallback chain work, and which failures should trigger the next model?
Why do you need a deadline as well as a per-attempt timeout?
Describe the three states of a circuit breaker and when it moves between them.
What is graceful degradation and what are two examples for an LLM feature?
Practice
Write first_working(models, call) that tries each model name in order using call(name) (which returns a string or raises RuntimeError) and returns (name, reply) for the first that works, or None. Test it with a call that fails for "a" and works for "b".
Write a Breaker class with failures and is_open(): it opens once failures reaches 2, and success() resets the count. Print is_open() after 1 failure, 2 failures, and after success().
Trade-offs
- ↔
A fallback model keeps the product up but may give weaker answers; measure and tell the user when it matters.
- ↔
A short cool-down recovers fast but probes often; a long one is gentle on the provider and slow to return to normal.
- ↔
More models in the chain increase resilience and the amount to test and keep updated.
Done when you can
Done when you can build a fallback chain that stops on non-retryable errors.
Done when you can enforce a deadline across several attempts.
Done when you can draw the breaker states and write the transitions.
Done when you can name two graceful degradations for your feature.
Done when you log which model answered and how often the breaker opens.