PHASE 13Advanced ~32 min· topic 3 of 8

Topic 13.3

Model Fallbacks and Degraded Modes

In one line

Providers have bad hours. A second model can answer when the first one fails in a way another model could fix, the screen must say which mode it is in, and a kill switch lets you turn the feature off in seconds with a useful page left behind.

Think of it like this

A restaurant when the main oven breaks. The kitchen can switch to the small oven and serve a shorter menu, and a sign at the door says so. If both ovens are down, the owner puts a note on the door with the phone number of the shop next street. Nobody is left standing outside without knowing what is happening.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Fallback: A second model (or a simpler path) used when the first fails.
Middleware: A function that wraps another one to add behaviour. In the AI SDK it wraps a model's calls.
Retryable error: A failure that may succeed if tried again or elsewhere: 429, 500, 502, 503. isRetryable on APICallError says so.
Circuit breaker: A guard that stops calling a failing service for a while, then tries one request to see if it recovered.
Kill switch: A flag that turns a feature off immediately, without a deploy.
Feature flag: A setting read at run time that turns a feature (or part of it) on or off for everyone or for some users.
Degraded mode: A reduced but still useful state: shorter answers, fewer features, or a plain help page.
503 Service Unavailable: The status for "not available right now". Your own kill-switch response uses it.

Step by step

01Wrap the model on the server

withFallback(first, second, onFallback) returns a model made by wrapLanguageModel. Its wrapStream tries doStream(). When that throws a retryable error it records that it fell back and returns second.doStream(params).

The route creates this wrapper per request, so the variable that remembers "we used the backup" belongs to one request only.

route.ts (wrapper)whole filets

import { wrapLanguageModel } from "ai";

function withFallback(first, second, onFallback) {
  return wrapLanguageModel({
    model: first,
    middleware: {
      wrapStream: async ({ doStream, params }) => {
        try {
          return await doStream();
        } catch (e) {
          if (!e.isRetryable) throw e; // our own mistake: do not hide it
          onFallback(e.statusCode);
          return second.doStream(params);
        }
      },
    },
  });
}
// in a real app: first = your main provider model, second = a cheaper or different one

02Tell the page which model answered

The first live example scripts four questions. The main model fails twice with 503, then works, then fails with 401. Questions 1 and 2 are answered by the backup and carry the "Backup mode" label. Question 3 is answered normally. Question 4 shows the isRetryable check at work: a 401 is not hidden by the backup, the server logs not recoverable: 401 and the page shows "Mitra could not answer right now."

The server's onError returns a short message for the page. Never return the provider's raw error text, which can contain internal details.

03Three screens from one error

The second example turns the same failure signal into three states. All fine: a normal answer. Both models down: Mitra is having trouble with three help links, the Ask button still usable so the user can retry. Kill switch on: Mitra is switched off for a short while, the button disabled, nothing sent (the fourth line).

The flag is read on every request, so flipping flags.aiEnabled takes effect on the next question with no deploy.

Three screens from one errordiagram

Rendering diagram…

04Do not hammer a dead provider

The third example is a circuit breaker. The main model fails on its first three calls. After two failures in a row the breaker opens and the next three requests go straight to the backup: the console shows the main model call count staying at 2. Request 6 is the trial: it fails (call 3), so the breaker opens again. Request 10 is the next trial: it works, the breaker closes, and requests 10 and 11 use the main model again.

Try it yourself

1
Take the backup down too
In the first example change backup to fakeModel({ fail: { status: 503 } }, quick). Question 1 now fails: the server logs the 503 from the backup too, and the page shows "Mitra could not answer right now." and no answer. Both models down is a real state. The second example below shows the screen to design for it.
2
Open the breaker sooner
In the third example change breaker.failures >= 2 to >= 1. Only 3 of the 11 requests reach the main model (requests 1, 5 and 9), and all three fail, so the main model never gets the fourth call that would have succeeded. A sensitive breaker protects the provider and the user's time, and a very sensitive one stays away from a model that had one bad moment.
3
Flip the switch twice
In the second example delete the line providersUp = true; before flags.aiEnabled = false;. The providers are still down when the switch goes on, and step 3 prints the same "switched off" panel as before: the flag is read first and no model is called. Notice no code is deployed: the next request just reads the new value.

Code & diagrams

A backup model, a label for it, and a 401 that is not hidden

The two models are scripted stand-ins. In a real app, `first` and `second` are two provider models (for example a larger and a smaller one). The one line that changes is where `main` and `backup` are created.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

(server) main model failed with 503 → using the backup
question 1 → 1 answers on screen, newest: Shipping takes 1 to 2 days. (Backup mode: shorter answers while the main model recovers) | error: none
(server) main model failed with 503 → using the backup
question 2 → 2 answers on screen, newest: Shipping takes 1 to 2 days. (Backup mode: shorter answers while the main model recovers) | error: none
question 3 → 3 answers on screen, newest: Orders ship in one to two working days, and you can track them in your account. | error: none
(server) not recoverable: 401
question 4 → 3 answers on screen, newest: Orders ship in one to two working days, and you can track them in your account. | error: Mitra could not answer right now.

Both models down, the kill switch, and a page that still helps

The 503 from the kill switch arrives as a normal response before any stream starts, so the page sees an error whose message is the JSON body. The `ai_paused` code is made up for this example.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

1 all fine → answers: 1 | panel: none | button disabled: false
2 both models down → answers: 1 | panel: Mitra is having trouble. You can try again, or use these pages. | button disabled: false
3 kill switch on → answers: 1 | panel: Mitra is switched off for a short while. | button disabled: true
4 click while paused → button is disabled, nothing sent

A circuit breaker: stop hammering a dead provider, then try once

Requests 3 to 5 and 7 to 9 never touch the main model: no wasted wait, no extra load on a provider that is struggling. Request 6 and 10 are the single trial requests.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

request 1 → Backup answer. [main failed with 503] | main model calls so far: 1
request 2 → Backup answer. [main failed with 503, breaker opened] | main model calls so far: 2
request 3 → Backup answer. [breaker open, main model not called] | main model calls so far: 2
request 4 → Backup answer. [breaker open, main model not called] | main model calls so far: 2
request 5 → Backup answer. [breaker open, main model not called] | main model calls so far: 2
request 6 → Backup answer. [main failed with 503, breaker opened] | main model calls so far: 3
request 7 → Backup answer. [breaker open, main model not called] | main model calls so far: 3
request 8 → Backup answer. [breaker open, main model not called] | main model calls so far: 3
request 9 → Backup answer. [breaker open, main model not called] | main model calls so far: 3
request 10 → Main model answer. [main] | main model calls so far: 4
request 11 → Main model answer. [main] | main model calls so far: 5

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Fall back on every error

In the first example delete the line if (!e.isRetryable) throw e; ... inside wrapStream.

terminal

$ Console, the last line

── what you'll see ──

(server) main model failed with 401 → using the backup

question 4 → 4 answers on screen, newest: Shipping takes 1 to 2 days. (Backup mode: shorter answers while the main model recovers) | error: none

Break #2

No breaker

In the third example change breaker.skip = 3; to breaker.skip = 0;.

terminal

$ Console, the last line

── what you'll see ──

request 11 → Main model answer. [main] | main model calls so far: 11

Myth vs fact

Myth

A fallback model gives the same answers.

Fact

A second model is a different model. Its answers can be shorter, different in style and sometimes worse. That is why the page labels the mode.

Myth

Retrying the same call is always enough.

Fact

The SDK already retries some failures (set maxRetries). A provider-wide outage needs a different model or a degraded page, not more retries of the same call.

Myth

A kill switch is for emergencies only.

Fact

It is also a rehearsal tool: turn it on in a quiet hour to check that the degraded page really works. A kill switch never tested is a guess.

Myth

Show the raw error so users can report it.

Fact

Raw provider errors can include internal details and mean nothing to a shopper. Show a human sentence, log the details on the server.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Fallback changes cost and privacy. A backup from a *different* provider means your users' text goes to another company: check your agreements and your privacy page before enabling it.
▸
Keep the fallback chain short (main, backup, help page). Long chains hide problems and multiply waiting time, since each failed attempt takes its own time to fail. A *slow* provider is a failure too: the SDK's timeout option (for example { firstChunkMs: 2000 }) ends a call that is too slow, but it aborts instead of falling back, so a race inside wrapStream is needed to switch to the backup (see the practice answer).
▸
Feature flags can be narrower than on/off: switch off only tools, only long inputs, or only a percentage of users while you investigate. Always have one flag that turns the whole feature off.
▸
Record every fallback and every breaker open in your logs as an event with a status code and no user text (Topic 13.5). A rising fallback rate is the earliest sign of trouble.

Remember this

1
The problem this solves: Your AI feature depends on a service you do not control. It will be slow, overloaded or down at some point, and sometimes it is *you* who must stop it (a bug that is costing money, a safety problem). Without a plan the user sees a spinner forever or a raw error. This topic builds the plan: a fallback model, honest labels, a circuit breaker and a kill switch.
2
A fallback is a second model used when the first one fails. Put it on the server, around the model call, so the page does not need to know. In the AI SDK you wrap a model with wrapLanguageModel and a middleware: a small object whose wrapStream function receives doStream (the real call) and the call's params. Try doStream(); if it throws, call the second model's doStream(params) and return that. The page receives an ordinary stream either way.
3
Fall back only for failures a second model can fix. Every APICallError has statusCode and isRetryable. A 429 (busy) and a 5xx (provider trouble) are retryable: another model may work. A 401 (bad key) or 400 (bad request) is *your* mistake: a backup would hide it, and you would only notice when the backup also fails or when the bill shows nobody uses your main model. Re-throw those so they show up in your alerts (first live example, question 4).
4
What this wrapper cannot catch: a stream that breaks after it started. The try only covers the moment doStream() is called, which is when a 429 or 503 arrives as a normal HTTP answer. If the connection drops after the model has already sent three words, the user has already seen those words and the error comes later, from inside the stream. Switching to the backup then would mean silently starting a different answer under text the user is reading. Treat it as a broken stream (Topic 3.3): keep what arrived, say it was cut off, and offer Retry, which goes through the fallback again.
5
Tell the user which mode they are in. A backup model may be cheaper, shorter, less accurate or lack a feature. Put the mode in the message metadata ({ mode: "backup" } on the finish part) and show a small label: "Backup mode: shorter answers while the main model recovers". Do not label it with a model name, which means nothing to a shopper. If a feature is *missing* in backup mode (for example, tools), disable its buttons instead of letting them fail.
6
A circuit breaker stops you from hammering a provider that is clearly down. After a few failures in a row it "opens": for a while the main model is not called at all and everyone goes straight to the backup. Then it lets one trial request through; if that works the breaker closes, if not it opens again (third live example). A real breaker counts seconds. The example counts requests so the output is exactly repeatable.
7
A kill switch is a flag, kept outside your code (an environment variable that can be changed without a redeploy, or a config service), that your route checks first. When it is off, the route answers 503 with a code such as ai_paused and calls no model. Use it when something is wrong *with you*: a runaway cost, a bad prompt shipped to everyone, a safety problem. Test it before you need it.
8
Degraded mode is a designed screen, not an error. Both models down, or the kill switch on, should lead to a page that still helps: links to the FAQ, shipping and returns pages, the contact page, and a clear sentence. The second live example shows three screens from one error: normal, "Mitra is having trouble" with a retry, and "Mitra is switched off" with the Ask button disabled.

Explain it without notes

How would you implement a model fallback with the AI SDK, and where does it live?

Which errors should trigger the fallback, and which should not?

What does a circuit breaker add on top of a fallback?

What is a kill switch for, and what must the user see when it is on?

Practice

Add a timeout to the first example's main model so a model that takes more than 200 ms to start is treated as failed. Which scripted option of the stand-in makes the main model slow?

Add a second flag backupEnabled to the second example. When it is false and the main model is down, the page should show the "trouble" panel immediately, without trying a backup.

Trade-offs

↔
A backup model keeps the feature alive and costs extra work: another provider account, another set of behaviours to test, another place user text goes. A static help page is cheaper and always works, but it cannot answer the question.
↔
Aggressive breakers protect users from long waits and may cut over to the weaker model too soon. Tune from your own failure data.

Done when you can

I can wrap a model with wrapLanguageModel so a retryable failure goes to a backup.
I can say which errors should not trigger a fallback and why.
I can show the user which mode answered and disable features missing in backup mode.
I can explain a circuit breaker and its one trial request.
I can build a kill switch and a degraded page that still helps.

Command Palette

Words you'll meet

Step by step

01Wrap the model on the server

02Tell the page which model answered

03Three screens from one error

04Do not hammer a dead provider

Try it yourself

Take the backup down too

Open the breaker sooner

Flip the switch twice

Code & diagrams

Break it on purpose

Fall back on every error

No breaker

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can