Python · Exercise 6 of 6 Previous
Intermediate
Score a model on an eval set
Write score_model(model, cases). Each case is a dict with "prompt" and "must_contain" (a list of words). A case passes if the model's reply contains every word, ignoring case. model is a function that takes a prompt string and returns the reply text. Return the share of passing cases as a number from 0.0 to 1.0. An empty list of cases gives 0.0. (Evals: Topic 16.11.)
Then write regressed(baseline, current, tolerance=0.05) that returns True when current is lower than baseline by more than tolerance.
What Check tests
- All cases pass gives 1.0
- Partial score
- Every word must be present
- Ignores case in both the reply and the required words
- Empty case list gives 0.0
- regressed uses the tolerance
Stuck? One hint at a time
The solution unlocks after your first Check. Try first: that is where the learning happens.
main.py
Sign in to run your code.
Output
Press Run to see what your program prints.