A weighted scorecard, two sets of weights
Quality, cost per 1,000 calls and p95 latency are made up. Replace them with your own eval results.
There is no best model, only the best model for your task, budget, speed, rules and data. Filter by hard requirements first, then test the shortlist on your own evals and compare with a weighted scorecard.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write reasons_out(candidate, need_context) that returns a list of reasons a candidate is out (an empty list means keep). Reasons: "context too small" when its context is below need_context, and "licence" when licence_ok is false.
Write lower_is_better_scores(values) that returns each value's score as 100 * min(values) / value, rounded to one decimal.
Explain it without notes
How do you choose a model for a new feature?
What is a gate, and why apply gates before running evals?
Why decide the scorecard weights before seeing the results?
Why is a leaderboard score not enough to choose a model?
Expected output
weights balanced: quality 0.5, cost 0.3, speed 0.2
small-open: quality 62, cost score 100.0, speed score 100.0, total 81.00
mid-hosted: quality 88, cost score 20.0, speed score 64.3, total 62.86
large-hosted: quality 93, cost score 4.4, speed score 34.6, total 54.76
winner: small-open
weights quality-first: quality 0.8, cost 0.1, speed 0.1
small-open: quality 62, cost score 100.0, speed score 100.0, total 69.60
mid-hosted: quality 88, cost score 20.0, speed score 64.3, total 78.83
large-hosted: quality 93, cost score 4.4, speed score 34.6, total 78.31
winner: mid-hosted