Topic 13.8
Playbook: Model Selection Scorecard
In one line
Five recipes for choosing and changing models with evidence: gates, the cheapest model that clears your quality bar, a weighted scorecard, a swap guard that compares models case by case, and configuration that makes switching a one-line change.
Think of it like this
A pilot's pre-flight checklist. The pilot does not rely on memory or feeling. They run the same short list every time, tick items, and refuse to take off if a critical item fails. These recipes are checklists for the model decision: the same steps every time, written down, runnable, and strict about the items that matter most.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Quality bar
- The pass rate on your eval set below which a model is not acceptable for this task.
- Critical case
- An eval case that must never regress, such as one that touches money or safety.
- Regression
- A case that the current model gets right and the new one gets wrong.
- Swap guard
- A check that compares an old and a new model case by case before a switch.
- Decision record
- A short note of the choice: date, candidates, eval version, scores, weights, winner and re-test date.
- Paired comparison
- Running two models on the same cases and comparing results case by case.
Step by step
01Step 1: write the gates and the quality bar
Gates come from rules and facts (context, data policy, licence). The quality bar comes from the cost of a wrong answer. Write both down before you run anything, in the same file as the eval set.
02Step 2: run the same eval on every candidate
Same prompt, same settings, same cases. The first example below uses scripted replies for three candidates, which is what the loop would collect from real calls. It prints the pass count and every failure, because "India" instead of "IN" is a formatting problem you may be able to fix with a better prompt, while "NP" for Nippon is a knowledge error.
03Step 3: pick by rule, not by taste
The rule here is: the cheapest candidate that passes at least the bar. In the example the small model fails the bar (4 of 6), the mid-size model passes (5 of 6) and the large one passes too but costs 4.5 times as much, so the mid-size model is chosen.
04Step 4: guard the switch
Before the chosen model replaces the current one, run the swap guard. It compares every case, lists improvements and regressions and blocks on any critical regression. The second example blocks a candidate that has the same overall score as the current model but breaks the critical case.
05Step 5: keep the call behind configuration
The real call differs only in base URL, model name and key (Topic 13.4). The code block shows the OpenAI-compatible SDK reading all three from the environment so the same program runs against a hosted API, a local server, or any model you are evaluating.
06Step 6: record, schedule, repeat
Save the decision record with the eval version and the date. Add a calendar item or a CI job to re-run the comparison when prices change or a new model appears (Topic 9.5). A choice that cannot be re-run cheaply will not be re-checked.
Try it yourself
- 1
Move the quality bar
In the first example set
QUALITY_BAR = 90. Predict who is chosen, then run. Set it to 60 and run again. What does this tell you about where the bar should come from? - 2
Fix the small model's format
Change the small model's replies to
["GB", "DE", "IN", "CI", "JP", "US"]as if a better prompt had fixed its formatting. Who is chosen now, and what is the saving? - 3
Make the guard pass
In the swap guard, change the candidate's first reply to
"GB"and its Ivory Coast reply to"XX". Predict the verdict, then run. Then setMAX_REGRESSIONS = 0and run again. - 4
Use the lab
Run the lab with the default model, then pick another lab model and run again. Write down tokens and time for each and decide as the first recipe would.
Code & diagrams
Scripted replies stand in for real calls. Prices are made up. The loop is the same with real calls.
Expected output
small-open: 4 of 6 passed (66 percent), cost 0.40 per 1k calls
failed: Bharat -> India
failed: U.S.A. -> US.
mid-hosted: 5 of 6 passed (83 percent), cost 2.00 per 1k calls
failed: Nippon -> NP
large-hosted: 6 of 6 passed (100 percent), cost 9.00 per 1k calls
chosen: mid-hosted
the most expensive candidate would cost 4.5 times as much per callBoth models pass 5 of 6, but the candidate regresses a critical case. Same average, different behaviour.
Expected output
current: 5 of 6, candidate: 5 of 6
improved: Nippon
regressed: United Kingdom
verdict: BLOCK (a critical case regressed: United Kingdom)Not run here: needs a reachable OpenAI-compatible server and your own key. Python and JavaScript use the openai package with a base URL (check its current docs for your version). Local OpenAI-compatible servers accept max_tokens; some newer hosted models expect max_completion_tokens instead, so check your provider. The Java sketch uses java.net.http.HttpClient and builds JSON as a string for brevity: use Jackson in a real project and escape the text.
import os
from openai import OpenAI
# The same code runs against any OpenAI-compatible server: only configuration changes.
client = OpenAI(
base_url=os.environ["LLM_BASE_URL"], # for example the hosted provider's URL, or a local server's
api_key=os.environ["LLM_API_KEY"],
)
MODEL = os.environ["LLM_MODEL"] # the candidate you are evaluating
def ask(text):
reply = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": text}],
temperature=0,
max_tokens=50,
)
return reply.choices[0].message.content
Try: Send the request to this model, then send the same request to the other lab model (change the model in the picker) and compare: did each reply in the exact format, how many tokens did each use, and how long did each take? Which would your quality bar and your budget choose?
Goals
- Codes in order: GB, DE, IN, CI, JP
- At most 10 words
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Skip the critical-case check
In the swap guard change if critical_regressed: to if False:.
Myth vs fact
Myth
If the average score is the same, the models are interchangeable.
Fact
They can fail on different cases. A swap guard compares case by case.
Myth
I should always pick the highest-scoring model.
Fact
Pick the cheapest model that clears the bar on your cases, within your gates. Extra quality you do not need is cost.
Myth
Choosing a model is a one-time decision.
Fact
Prices, models and your traffic change. Keep the harness so the decision can be re-run in minutes.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Run the swap guard in CI (Topic 9.5) against recorded or live calls, with the critical cases as a hard gate. A model change then goes through the same review as a code change.
- ▸
Run a shadow period: send real traffic to the candidate but keep serving the current model, and compare the answers offline before switching (Topic 9.7).
- ▸
Release gradually: a small percentage of traffic first, the same metrics as before (Topic 11.4), a one-line rollback to the previous model name from configuration (Topic 12.7).
- ▸
Track cost per successful task, not cost per call, after the switch. A cheaper model that needs more retries or repairs (Topic 4.6) can erase the saving.
Remember this
- 1
Recipe: Gates, then a shortlist. When to use it: before any testing, whenever a new model is proposed. It guards against spending evals and meetings on models you could never use (too small a context window, forbidden data flow, wrong licence). How to adapt it: write each gate as a function that returns reasons, so the decision is explainable. Topic 13.1 has the runnable version.
- 2
Recipe: The cheapest model that clears the bar. When to use it: a task with a clear pass or fail on your eval set. It guards against paying for the strongest model by default. How to adapt it: set the quality bar from the cost of mistakes (a higher bar for money, a lower one for drafts), run every candidate on the same cases with the same settings, then choose the cheapest candidate at or above the bar. The first example below does exactly that and prints what the failures were, which matters as much as the percentage.
- 3
Recipe: A weighted scorecard with a sensitivity check. When to use it: when quality, cost and speed all matter and the cheapest-above-the-bar rule is too blunt. It guards against choosing on one number. How to adapt it: fix the weights before you see results, put every measure on a 0 to 100 scale in the same direction, and re-run with weights moved a little. If the winner flips, tell the owner of the product that this is a priorities question. Topic 13.1 has the runnable version.
- 4
Recipe: The swap guard. When to use it: every time a model, a prompt or a provider version changes in production. It guards against the quiet regression where the average stays the same and behaviour does not. How to adapt it: compare current and candidate case by case, list improvements and regressions, mark cases that must never regress (critical), block on any critical regression and cap the number of other regressions. The second example shows two models that both score 5 of 6 where the candidate breaks the one critical case.
- 5
Recipe: Models as configuration, with an expiry date. When to use it: always. It guards against being locked to one model and against decisions that quietly go stale. How to adapt it: keep the base URL, the model name and the key in environment variables or a config file (Topic 12.7), put the model behind one interface (Topic 2.8), store the decision record next to the eval results, and schedule a re-test whenever prices or models change. The code block at the end shows the one place where the real call lives.
- 6
Before you ship a model choice. The checklist at the end of this topic is the phase in one page. The single most useful habit: never change a model without running the same eval harness on the old and the new one and reading the case-by-case differences.
Explain it without notes
Describe a repeatable process for choosing a model for a new feature.
Why can two models with the same pass rate still not be interchangeable, and how do you protect against it?
How should model names and providers appear in your code?
Practice
Write pass_percent(replies, expected) that returns the whole-number percentage of replies equal to the expected value after strip() and upper(), and print it for three replies with one failure.
Write regressions(before, after) where each is a list of booleans (passed or not, per case); it returns the indexes where before is true and after is false. Print the result for [True, True, False, True] and [True, False, True, True].
Trade-offs
- ↔
A strict swap guard blocks risky changes but slows releases; a short critical list keeps it fast.
- ↔
A cheapest-above-the-bar rule is simple and explainable; a scorecard handles trade-offs but needs agreed weights.
- ↔
Shadow runs give real evidence and cost double the calls for a while.
Done when you can
Done when gates, quality bar and weights are written down before the eval runs.
Done when every candidate was tested on the same cases with identical prompts and settings.
Done when the choice follows a rule and the failures of each candidate were read, not just the percentages.
Done when a swap guard with critical cases passed before the switch.
Done when model name, base URL and key come from configuration and rollback is one change.
Done when the decision record and a re-test date are saved.