Topic 12.6
Model Routing
In one line
Not every request needs your most expensive model. A router sends easy requests to a cheap, fast model and escalates hard ones to a stronger model, using a confidence or validation signal. Measured on your own examples, it can cut cost a lot, but only if the escalation signal is trustworthy.
Think of it like this
A hospital. A nurse at the front desk handles a sprained finger. A tricky case is passed to a doctor, and a rare one to a specialist. Everyone is paid according to their level, and the specialist's time is kept for cases that need it. Sending every patient to the specialist would be safe and absurdly expensive. Model routing is the same triage.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Router
- Code that chooses which model handles each request.
- Cascade
- Trying a cheap model first and moving to a stronger one only if needed.
- Escalate
- Pass a request on to a stronger (and costlier) model.
- Confidence
- A number saying how sure an answer is; from models it is often unreliable.
- Threshold
- The cut-off value: below it, escalate.
- Silent miss
- A wrong answer that was not escalated because the cheap model sounded sure.
Step by step
01Step 1: the baseline
Always using the large model is the simplest design and the most expensive one. Write down its cost for your real traffic first. It is the number the router has to beat.
02Step 2: pick a signal
The example gives each task a confidence from the small model (in a real system this comes from validation, a score or a check). A threshold of 0.7 means answers below it are escalated.
03Step 3: write the cascade
Every task pays for the small call. If confidence is under the threshold it also pays for the large call. The code keeps the running totals so the savings are computed, not guessed.
04Step 4: read the numbers
Two of six tasks escalate (t2 and t5). They cost the small price plus the large price, so they are more expensive than the large model alone. The rest cost one twentieth. The total saving is shown as a percent of the always-large cost.
05Step 5: find the break-even
With these prices a task that escalates costs 1.05 times as much as always-large (small is a twentieth of large). So the router loses on escalated requests and wins on the others; it pays off whenever fewer than about 95 percent escalate. Change the prices and thresholds in the example and watch the saving move.
06Step 6: measure quality, not just cost
The lab below sends the same task to a small and a larger model so you can compare answers, tokens and time. In production you do this on a golden set and decide where the threshold goes.
Try it yourself
- 1
Move the threshold
Change
THRESHOLDto 0.9 and predict how many tasks escalate before running. What happens to the saving? - 2
Make the small model pricier
Set the small price to
(1.00, 4.00). Does the router still save money? At what escalation share does it stop saving? - 3
Run the lab twice
In the lab, run the small model, then the larger one. Note tokens and time for each. How many seconds does an escalation add if you call them one after the other?
Code & diagrams
Example prices held in variables; the small model's confidence decides who answers. Check real prices with your provider.
Expected output
t1: confidence 0.90 -> small, cost 0.000040
t2: confidence 0.50 -> small -> large, cost 0.001302
t3: confidence 0.95 -> small, cost 0.000031
t4: confidence 0.80 -> small, cost 0.000049
t5: confidence 0.60 -> small -> large, cost 0.001680
t6: confidence 0.85 -> small, cost 0.000030
escalated 2 of 6
routed 0.003132 dollars, always large 0.005840 dollars
saved 46.4 percentTry: Send this request to openai/gpt-oss-20b, then change the model to openai/gpt-oss-120b and send again. Compare the answers, the completion tokens and the time. Would the small model's answer have been good enough to skip escalation?
Goals
- The reply finished on its own
- The final price 214.2 appears
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Escalate on the wrong side
Flip the comparison so that the large model is called when the small model is confident: if confidence >= THRESHOLD: (JavaScript and Java the same).
Myth vs fact
Myth
Routing always saves money.
Fact
Escalated requests pay twice and add latency. It helps when most requests are easy and the signal is reliable.
Myth
A model's own confidence is a reliable escalation signal.
Fact
Self-reported confidence is often poorly calibrated. Prefer checks your code can run: schema validation, tests, agreement, retrieval support.
Myth
The large model is always better.
Fact
Often for hard reasoning, but not always for narrow, well-defined tasks, and it is slower. Measure on your own evals.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the route for every request: model, signal value, escalated or not, tokens, cost and latency. Sample escalated and non-escalated answers for human review to find silent misses (Topic 9.7).
- ▸
Latency budgets matter: in a cascade the user waits for the small call and then the large one. For interactive use, set a deadline and escalate only if time remains, or start both and take the first good answer (at a cost).
- ▸
Keep a per-feature route table in configuration so a product team can move a feature to a different tier without a deploy (Topic 12.7), and run the eval in CI when it changes.
- ▸
Providers change models and prices. Keep your price table in configuration with a date, and recompute the saving regularly.
Remember this
- 1
Why this exists. Model prices and speeds differ by large factors between small and large models (Topic 6.1, Phase 13). Many production requests are easy: a short classification, a simple lookup, a friendly greeting. Paying large-model prices for them wastes money and makes users wait longer. Yet a few requests are hard and need the stronger model. A router matches the model to the request.
- 2
Route by task first. The simplest router is a table: classification and extraction go to the small model, long reasoning and code review go to the large one, based on the feature rather than the individual request. It needs no extra model call, is easy to explain and test, and is where to begin.
- 3
Cascade: cheap first, escalate when needed. Call the small model. Look at a signal that says whether its answer can be trusted. If yes, return it. If no, call the large model. Recall from Topic 4.5 that validation is code that checks the output's shape. Useful signals: output that fails validation (invalid JSON, wrong schema, Topic 4.5), an explicit "I am not sure" or low self-reported confidence (models are often poorly calibrated, meaning a stated 90 percent does not come true 90 percent of the time, so treat it as weak), disagreement between two cheap samples, a refusal, or a long or complicated input. Validation failures are the most reliable signal because your code can check them.
- 4
The cost maths. Let
pbe the share of requests that escalate. Each escalated request pays for the small call and the large call. Expected cost per request issmall + p x large. Compared with always using the large model (large), you save whenpis below1 - small/large. With the example prices, small costs one twentieth of large, so the router saves money until nearly all requests escalate. The saving shrinks if the small call is slow, because escalation also adds latency (two calls one after the other). - 5
Quality is the other half. Cost saved is only good if answers stay as good. Measure on a golden set (Topic 9.2): the quality score of always-large, always-small and the router, plus cost and latency. The risk is a silent miss: the small model is confidently wrong, nothing escalates, and the user gets a wrong answer. Check these cases in the eval and tune the threshold with data, not guesses.
- 6
Alternatives and refinements. A tiny classifier (or a small model) can predict difficulty before any answer is generated, which avoids the wasted small call but adds its own cost and errors. Use the cheaper model for steps inside a pipeline (query rewriting, classification) and the stronger one for the final answer. Add a fallback for outages (Topic 6.9): routing by availability is a different reason than routing by difficulty, so log which one happened.
- 7
Operate it. Put the thresholds and model names in configuration (Topic 12.7), log the route taken, the signal and the cost for every request (Topic 11.4), watch the escalation rate over time (a sudden rise can mean a provider change or a new kind of user input), and re-run the evals whenever a model changes.
Explain it without notes
What is a model cascade and when does it save money?
What signals can trigger escalation, and which are most reliable?
How do you evaluate a router before using it in production?
What new failure modes does routing add?
Practice
Write route(confidence, threshold) returning "small" when the confidence is at least the threshold and "large" otherwise, and print the route for the confidences 0.9, 0.7 and 0.5 with a threshold of 0.7.
Write expected_cost(small, large, p) returning small + p * large and print it for p of 0.1 and 0.3 with small 0.001 and large 0.02 (format with 4 decimals).
Trade-offs
- ↔
A cascade saves money on easy traffic and adds latency and complexity on hard traffic.
- ↔
A task-based route table is simple and predictable; a per-request router adapts better and is harder to test.
- ↔
Tight thresholds protect quality and give up savings; loose ones save more and let more silent misses through.
Done when you can
Done when you can explain a cascade and compute its expected cost.
Done when you can name reliable escalation signals and a weak one.
Done when you can say how to evaluate a router on quality, cost and latency.
Done when you can name the failure modes routing adds.
Done when you can keep thresholds and model names in configuration.