Topic 13.1
Choosing a Model
In one line
There is no best model, only the best model for your task, budget, speed, rules and data. Filter by hard requirements first, then test the shortlist on your own evals and compare with a weighted scorecard.
Think of it like this
Choosing a van for a delivery business. You do not pick it from the top speed in a magazine. First you write the must-haves (the boxes fit, it is allowed in the city centre). Then you test-drive two or three vans on your own route with your own boxes. Only then do you weigh price, fuel and comfort by what matters to your business. Choosing a model is the same job: must-haves, test drive, weighing.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Candidate
- A model you are considering, with the facts and numbers you collected about it.
- Gate
- A hard requirement. A model that fails it is out, whatever its score.
- Eval set
- Your own collection of real test cases with expected answers (Topic 9.2).
- Scorecard
- A table that puts every measure on one scale, weights them and adds them up to one total per candidate.
- Normalise
- Convert different units (dollars, milliseconds, percent) to the same 0 to 100 scale so they can be added.
- Context window
- The most tokens one call can hold, input plus output together.
- Data policy
- A provider's promises about where your prompts go, how long they are kept and whether they train on them.
- Open weights
- A model whose trained numbers you can download and run yourself (Topic 13.2).
Step by step
01Step 1: write the requirements before looking at models
Write down, in plain sentences, what the feature needs: the longest prompt and reply in tokens, the rules about customer data, the budget per thousand calls, the slowest acceptable reply. This list stops you from falling in love with a model and then bending the requirements to fit it.
REQUIREMENTS = {
"max_prompt_plus_reply_tokens": 6000,
"customer_text_must_stay_ours": True,
"budget_dollars_per_1k_calls": 3.00,
"p95_latency_ms": 2000,
"min_pass_rate_on_our_evals": 0.85,
}02Step 2: apply the gates
Each gate is a yes/no check that gives a reason. The first example runs five made-up candidates through three gates and prints why each one stays or goes. Nothing here needs a model call: it is a list and three ifs.
03Step 3: run the same evals on every candidate
Same prompt, same settings, same cases, for every candidate. Record pass rate, tokens in and out, and time per call. The numbers in this phase's examples are made up so that the code is repeatable, but the real run is just a loop over your golden set (Topic 9.8).
04Step 4: put every measure on the same scale
Quality is already 0 to 100. Cost and latency are "lower is better", so the cheapest candidate gets 100 and the others get cheapest divided by theirs, times 100. A model that costs 5 times more gets a cost score of 20.
costs = [0.40, 2.00, 9.00]
scores = [100 * min(costs) / c for c in costs]
print([round(s, 1) for s in scores]) # [100.0, 20.0, 4.4]05Step 5: weigh, add up, then try other weights
The second example scores three candidates with two sets of weights. With balanced weights the cheap, fast model wins. With quality-first weights a larger model wins. The data did not change, only your priorities, and that is exactly the conversation to have with the people who own the product.
06Step 6: record the decision and the expiry
Write down the chosen model, the date, the eval-set version, the scores and the weights. When a new model is released or a price changes, re-run the same harness. Because the decision is a script and a table, re-deciding takes minutes, not a meeting.
Try it yourself
- 1
Move a gate
In the first example change
need_contextto 20000. Predict who is left on the shortlist, then run. Then setneed_data_stays_ourstoFalseand run again. Which gate removed the most candidates? - 2
Flip the winner
In the scorecard try the weights 0.6, 0.2, 0.2. Does the winner change from the balanced run? Search for the weight on quality at which the winner flips between the small and the mid-size candidate.
- 3
Use the labs
Run both labs. Write down tokens used and time for each. Are the answers the same? If they are, how much more would the larger model cost you per thousand calls at your provider's prices, and is that worth it?
Code & diagrams
Candidates are made-up labels. The gates are context, data policy and licence.
Expected output
tiny-open: out (context too small)
small-open: keep
mid-hosted: keep
large-hosted: out (data leaves our control)
odd-licence: out (licence does not allow our use)
shortlist: small-open, mid-hostedQuality, cost per 1,000 calls and p95 latency are made up. Replace them with your own eval results.
Expected output
weights balanced: quality 0.5, cost 0.3, speed 0.2
small-open: quality 62, cost score 100.0, speed score 100.0, total 81.00
mid-hosted: quality 88, cost score 20.0, speed score 64.3, total 62.86
large-hosted: quality 93, cost score 4.4, speed score 34.6, total 54.76
winner: small-open
weights quality-first: quality 0.8, cost 0.1, speed 0.1
small-open: quality 62, cost score 100.0, speed score 100.0, total 69.60
mid-hosted: quality 88, cost score 20.0, speed score 64.3, total 78.83
large-hosted: quality 93, cost score 4.4, speed score 34.6, total 78.31
winner: mid-hostedTry: Send the request and read the answer, the token usage and the time. Then open the next lab (a larger model, same request) and compare: did both label all four messages correctly? Which used fewer tokens? Which was faster?
Goals
- Labels in order: billing, bug, billing, other
- At most 15 words
- The reply finished on its own
Sign in to send this request to a real model (free).
Try: Send the same request to the larger model. Compare tokens used (including hidden reasoning) and total time with the previous lab. If both answers are right, the cheaper and faster model wins on this task.
Goals
- Labels in order: billing, bug, billing, other
- At most 15 words
- The reply finished on its own
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Use raw numbers as scores
In the scorecard example replace the two normalised scores by the raw numbers: cost_score = c["cost_per_1k"] and speed_score = c["p95_ms"].
Myth vs fact
Myth
The model at the top of the leaderboard is the best choice.
Fact
A leaderboard measures someone else's tasks. The best model for you is the cheapest one that passes your evals, within your gates.
Myth
Bigger models are always better.
Fact
Bigger models are usually stronger on hard reasoning, but for narrow, well-specified tasks a small model often passes the same tests at a fraction of the cost and latency.
Myth
Once I pick a model I am done.
Fact
Models are retired, updated and replaced by cheaper ones. Keep the eval harness and re-run it when anything changes.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Choose with paired comparisons on the same cases, not two unrelated averages. The same 50 cases with different models let you list the cases where model A passes and B fails, which is what you actually want to read (Topic 13.8).
- ▸
Measure cost per successful task, not cost per call. A cheap model that needs a retry on 30% of tasks, or a repair loop (Topic 4.6), can cost more than a stronger model that passes first time.
- ▸
Check the effective context window on your own long inputs: a model can accept a long prompt and still use the middle of it badly (Topic 1.5). Run one eval that puts the needed fact in the middle.
- ▸
Keep a written decision record: date, candidates, eval-set version, scores, weights, the winner and why. In six months it explains the choice and tells you what to re-test.
Remember this
- 1
Why this exists. There are dozens of models, they change every few months, and "the best model" means nothing without a task. Picking by buzz leads to two expensive mistakes: paying for a large model when a small one passes your tests, or shipping a cheap model that fails quietly on your real inputs (Topic 0.4). A repeatable method is cheaper than a favourite and survives the next model release.
- 2
Six things to compare. (1) Quality on your task, measured by you. (2) Cost per call, from input and output tokens and prices (Topic 6.1). (3) Latency: time to first token and total time (Topic 6.3). (4) Context window: the most tokens, input plus output, one call can hold (Topic 1.5). (5) Licence and terms: what you are allowed to do with the model and its outputs. (6) Data policy: where prompts go, how long they are kept, and whether they may be used for training. Some of these are gates, the rest are trade-offs.
- 3
Gates first. A gate is a hard requirement: if a model fails it, it is out, however good it is. Your longest prompt does not fit its context window: out. Your rules say customer text must not leave your control, and the model is only offered as a remote service: out. The licence forbids your use: out. Gates need no testing, cost nothing, and shrink the list before you spend any money on evals. Write each gate as a yes/no check with a reason, so the decision is explainable later.
- 4
Then test on your own evals. Take your golden set (Topic 9.2), 30 to 100 real cases, and run every shortlisted model with the same prompt and settings. Record the pass rate, the tokens used and the time taken. Public rankings measure other tasks, often with contaminated test sets (Topic 13.7). Your 50 cases measure the thing you are shipping. A free lab at the end of this topic lets you do a small version of this with two real models.
- 5
A weighted scorecard turns the numbers into a decision. Put each measure on the same 0 to 100 scale (for "lower is better" measures such as cost and latency, give the best candidate 100 and scale the others by best divided by theirs). Give each measure a weight, the weights sum to 1, and the total is the sum of weight times score. Decide the weights before you see the results, or you will tune them until your favourite wins. Then change the weights a little: if the winner flips, the decision is sensitive and you need a human conversation about priorities, not more arithmetic.
- 6
The answer has an expiry date. Prices drop, new models arrive, providers retire old ones. Keep the model behind an interface (Topic 2.8) and the name in configuration (Topic 12.7), keep your eval set in the repository, and re-run the comparison when something changes. Start with the smallest model class that passes your quality bar; Topic 12.6 shows how to escalate to a larger one only when needed.
Explain it without notes
How do you choose a model for a new feature?
What is a gate, and why apply gates before running evals?
Why decide the scorecard weights before seeing the results?
Why is a leaderboard score not enough to choose a model?
Practice
Write reasons_out(candidate, need_context) that returns a list of reasons a candidate is out (an empty list means keep). Reasons: "context too small" when its context is below need_context, and "licence" when licence_ok is false.
Write lower_is_better_scores(values) that returns each value's score as 100 * min(values) / value, rounded to one decimal.
Trade-offs
- ↔
More candidates give a better chance of finding a good fit but multiply eval time and cost; use gates and a first quick pass to keep the list short.
- ↔
Many weighted measures capture more of reality but make the weights easier to game; keep the scorecard to three to five measures.
- ↔
A small eval set is cheap to run but noisy (Topic 13.7); a larger one costs more per comparison but tells you which differences are real.
Done when you can
Done when you can write requirements and turn them into gates with reasons.
Done when you can run your own eval set on several candidates with identical settings.
Done when you can put cost, latency and quality on one 0 to 100 scale and apply weights.
Done when you can show how the winner changes with the weights.
Done when you can write a decision record with a date to re-test.