PHASE 13Advanced ~38 min· topic 8 of 8

Topic 13.8

Build: Mitra in Production

In one line

Mitra gets a usage meter, a daily limit enforced on the server, a backup model, a cache for general questions, a kill switch, content-free metrics and a test suite. It can now run on a bad day without surprising the user or the bill.

Think of it like this

The night before the shop opens to the public. The till is counted, the sign on the door has opening hours, there is a spare key, a fire exit, and a note for the staff about what to do if the power fails. The shop looked fine to the owner yesterday. Tonight you check what happens when things go wrong.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Production layer: The code around the feature that handles limits, failures, cost and measurement, and is invisible when things go well.
Usage meter: A small display of how much of the allowance is left.
Backup mode: The state where a second model answers because the main one failed.
Kill switch: A server flag that turns the feature off at once.
Cache hit: An answer returned from saved answers, without calling the model.
Test suite: The set of tests that run together to check the feature still works.
Metrics: Counters of what happened (answered, failed, stopped, backup), kept without any of the words people typed.

Step by step

01The route as a short list of decisions

The first example calls the route the way the browser does and prints the status and the quota header for nine requests. Read it as a story. Asha's return-window question is a miss (the model runs and the answer is saved), and Ravi's differently spelled version is a hit (quota used 0, nothing counted). Personal questions are never cached, so Ravi's "Where is my order?" asks the model again. Asha's fourth counted question gets a 429 with retry-after 3600. During the outage the stream still answers and says "mode":"backup". With the kill switch on, the answer is 503 {"error":"ai_paused"}.

The route as a short list of decisionsdiagram

Rendering diagram…

02What the route would look like with a real provider

In a real project the route is a file in your app, the models come from provider packages, and the counters and the cache are in Redis or a database. The shape of the code does not change. The model ids are placeholders: use your provider's current ones.

app/api/chat/route.ts (not run here)whole filets

import { convertToModelMessages, streamText, wrapLanguageModel } from "ai";
import { mainModel, backupModel } from "@/lib/models";        // e.g. provider("provider/main-model"), provider("provider/backup-model")
import { flags, quota, cache } from "@/lib/production";      // Redis or database behind these

export async function POST(req: Request) {
  const user = await currentUserId(req);                      // from your session, never from a header the page sets
  if (!(await flags.get("aiEnabled"))) return Response.json({ error: "ai_paused" }, { status: 503 });

  const { messages } = await req.json();
  const question = lastUserText(messages);
  const personal = isPersonal(question);                      // decided from how the request is built, not only from words

  if (!personal) {
    const hit = await cache.get(keyFor(question));
    if (hit) return streamSaved(hit, await quota.headers(user)); // free: not counted
  }

  const allowed = await quota.tryUse(user);                   // check + count in ONE atomic step
  if (!allowed.ok) return Response.json({ error: "daily_limit" }, { status: 429, headers: { "retry-after": String(allowed.retryAfter) } });

  let mode: "main" | "backup" = "main";
  const model = wrapLanguageModel({
    model: mainModel,
    middleware: { wrapStream: async ({ doStream, params }) => {
      try { return await doStream(); }
      catch (e: any) { if (!e.isRetryable) throw e; mode = "backup"; return backupModel.doStream(params); }
    } },
  });

  const result = streamText({
    model,
    messages: await convertToModelMessages(messages),
    maxOutputTokens: 600,                                     // a cap per answer, from your own budget
    onEnd: async (e) => {
      await usage.record({ user, input: e.usage.inputTokens, output: e.usage.outputTokens, finish: e.finishReason, mode });
      if (!personal && mode === "main" && e.finishReason === "stop") await cache.set(keyFor(question), e.text);
    },
  });
  return result.toUIMessageStreamResponse({
    headers: await quota.headers(user),
    messageMetadata: ({ part }) => (part.type === "finish" ? { mode } : undefined),
    onError: () => "failed",
  });
}

03The page and the suite

The second example is Mitra's chat with the production layer, and it does two things when it loads. First it runs a five-test suite with the Topic 13.6 helpers: remaining questions are shown; a repeated general question is a free cache hit; at the limit the box is locked and the wait is shown; during an outage the backup answers and the page says so; with the kill switch on the paused panel appears. Each test uses its own user id so they do not share counters, and waits for what appears on screen.

Then it renders the real Mitra and a scripted visitor asks four questions. The meter reads "2 of 3", "2 of 3" (the cached general question is free), "1 of 3", "0 of 3", and the box locks with "Try again in 1 hour." The last line prints the counters: four answered, none failed, stopped or from the backup.

04Rehearse the bad day

Before launch, flip each lever on purpose: turn the kill switch on and look at the page, set the main model to fail and look for the label, set a user's count to the limit and read the message. These are the tests above, and doing them once by hand in a staging environment finds the problems that tests, written by the same person, miss.

Try it yourself

1
Rehearse an outage
In the second example add the line mainUp = false; just before the line createRoot(document.getElementById("root")).render(<Mitra user="asha" />); (after the suite has run). The three personal answers now carry the "Backup mode" label and the metrics end with "backup":3. The return-window question is still a free cache hit: a saved answer needs no model, so an outage cannot touch it. The meter still counts the three backup answers.
2
Lower the limit
Set LIMIT to 2 (it is one constant used by the server and the page). The visitor locks sooner, and the first three suite tests fail: they expect the text "2 of 3", "3 of 3" and "all 3 questions", written by hand. A test that hard-codes a number you may change breaks when you change it. Make the expectations read LIMIT and they follow.
3
Add a failing test
Add a test that expects "1 of 3 questions left today" after a single personal question. It fails with "Unable to find an element with the text…". Fix the expectation, not the app, and notice that you needed no timer to learn it.

Code & diagrams

Mitra's production route: nine requests, read as a story

Everything here is a stand-in: the models are scripted, the counters are a Map, the user id comes from a header. In a real route the user id comes from the session, and the counters live in a shared store.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

asha  "What is the return window?"       → 200 quota used 1 of 3
ravi  " what is the RETURN window "      → 200 quota used 0 of 3
asha  "Where is my order?"               → 200 quota used 2 of 3
ravi  "Where is my order?"               → 200 quota used 1 of 3
asha  "Where is my order ORD-1042?"      → 200 quota used 3 of 3
asha  "Where is my order ORD-1042?"      → 429 retry-after 3600
ravi  "Where is my order?"               → 200 quota used 2 of 3
during the outage the stream says it came from the backup model: true
ravi  "Hello"                            → 503 {"error":"ai_paused"}

Mitra in production: the page, a test suite, and a scripted visitor

The suite runs first (its components are removed after each test), then the real Mitra mounts and a scripted visitor uses it. The visitor's second question is free because a test already cached the answer. Type your own questions: the fourth counted one locks the box.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

PASS a personal answer shows the questions left
PASS a repeated general question is served from cache and costs nothing
PASS at the limit the box is locked and the wait is shown
PASS when the main model is down, the backup answers and the page says so
PASS with the kill switch on, Mitra shows the paused panel
asha asked "Where is my order?" → Order ORD-1042 (Masala chai 250 g) has shipped. | 2 of 3 questions left today
asha asked "What is the return window?" → Returns are accepted within 7 days of delivery. | 2 of 3 questions left today
asha asked "Where is my order ORD-1042?" → Order ORD-1042 (Masala chai 250 g) has shipped. | 1 of 3 questions left today
asha asked "Is my parcel here yet, my order?" → Order ORD-1042 (Masala chai 250 g) has shipped. | 0 of 3 questions left today
at the limit: You have used all 3 questions for today. Try again in 1 hour. | box disabled: true
metrics (counts only, no text): {"answered":4,"failed":0,"stopped":0,"backup":0}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Let personal answers into the cache

In the first example change const personal = isPersonal(question); to const personal = false;.

terminal

$ Console, the Ravi lines

── what you'll see ──

ravi "Where is my order?" → 200 quota used 0 of 3

Myth vs fact

Myth

Production readiness is something you add at the end.

Fact

Limits, fallbacks and tests change the design of the route and the page. They are easiest to build while the feature is small.

Myth

If the demo works, the feature works.

Fact

A demo shows the good path. Production is made of the other paths, and this phase scripts each of them.

Myth

The meter is decoration.

Fact

It turns a surprising refusal into an expected one. People accept limits they can see coming.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep the production layer in its own module with a tiny interface (flags, quota, cache, usage). Then the route reads like the diagram and each part is testable alone.
▸
Put a monthly spend alarm outside the app (in your provider's dashboard or billing tool) as well as in it. If your code is the thing that is broken, it cannot protect you.
▸
Run the offline rehearsal (kill switch on, main model failing, limit reached) in staging before every launch and write down what the user saw.
▸
Review the logs once with a fresh pair of eyes and ask: could this line identify a person? If yes, remove it.

Remember this

1
The problem this solves: Phases 0 to 12 made Mitra work well when everything works. This topic puts together the production pieces of Phase 13 so Mitra copes with limits, outages, repeated questions and your own mistakes, and so you can prove it with tests.
2
Starting point. Mitra is a chat on the AI SDK: a /api/chat route with streamText, a useChat client with Stop, safe formatting, cards, tools with approval and defences against hostile input (Phases 5 to 9), and the product touches, files and voice, and in-browser model of Phases 10 to 12. To keep this example readable it shows only the chat itself in its simplest form: the production layer is what is new, and it wraps the features you already have.
3
The route, in order. (1) The kill switch: if the flag is off, answer 503 ai_paused and call nothing. (2) General question already in the cache: stream it back for free (a hit does not count against the quota). (3) Quota: 3 counted questions per user per day (a made-up number), checked and counted in one step, 429 with retry-after when over. (4) The model, wrapped with a backup for retryable failures, with the mode and token usage sent as message metadata. (5) On a normal stop, a general answer from the full-quality model is saved in the cache. Personal questions are never cached.
4
The page, in order. A meter ("2 of 3 questions left today") fed by response headers. A "Backup mode" label when the metadata says so. At the limit the box is locked and the wait is explained. When Mitra is switched off, a panel replaces the chat and points to the help pages. Stop works as in Phase 3. Errors have a person-sized sentence. Every finished request adds one to a counter (answered, failed, stopped, backup), and no text is ever recorded (Topic 13.5).
5
What Mitra can do now that it could not before: show how many questions are left; refuse politely at the limit without calling the model; keep answering through a provider failure with a labelled backup; answer repeated general questions instantly and for free; be switched off in seconds; count its own outcomes without recording what anyone typed; and prove all of this with a test suite that waits on the screen.
6
What is still missing, and which phase adds it: Mitra answers one request at a time. It cannot plan several steps, run for minutes, be resumed after the user leaves, be steered mid-task, or show other agents: Phase 14 (Agents in the UI) adds those, with the multi-step refund as its build. Phase 15 assembles everything into the capstone and walks through interview questions, a design exercise and code review. Real prices, a real database for the counters and real monitoring are yours to connect: this phase showed the shape, with stand-ins.

Explain it without notes

Walk through what Mitra's route does for one request, in order.

Which production pieces are on the server, which on the page, and why?

How do you prove the production behaviour works?

What is still missing from Mitra after this phase?

Practice

Add a monthly counter to the route's used logic: a second Map usedMonth with a limit of 10, checked together with the daily one. Which message should the user see when only the monthly limit is hit?

Add a test that a general question asked in backup mode is not cached.

Trade-offs

↔
Every production piece adds code and states to test. Add them in the order of risk: a budget cap and a kill switch first, then the limit, the fallback, the cache and the measurements.
↔
Showing the limit and the mode makes the product more honest and a little busier. For most users a quiet meter and a small label are the right amount.

Done when you can

I can describe the order of decisions in a production chat route.
I can show the meter, the near-limit warning, the locked box and the paused panel from server data.
I can label backup mode and keep backup and personal answers out of the cache.
I can run a test suite that scripts the model and waits on the screen.
I can say what Mitra still lacks and which phase adds it.

Command Palette

Words you'll meet

Step by step

01The route as a short list of decisions

02What the route would look like with a real provider

03The page and the suite

04Rehearse the bad day

Try it yourself

Rehearse an outage

Lower the limit

Add a failing test

Code & diagrams

Break it on purpose

Let personal answers into the cache

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can