LLM Engineering · Exercise 5 of 5 Previous
Intermediate
Score a prompt version and gate the change
Write score(answers, golden) that takes a dictionary of the model's answers by case id and a dictionary of expected labels by case id, and returns the integer percentage of cases answered correctly (right * 100 // total). Ignore leading and trailing spaces and letter case in the answers. A missing answer is wrong.
Write gate(base_pct, pct, min_percent=90, max_drop=5) that returns "PASS", "BLOCK: under the bar" when pct is below min_percent, or "BLOCK: dropped too far" when pct is more than max_drop points below base_pct. Check the bar first.
What Check tests
scoretolerates spaces and case- A wrong or missing answer counts against the score
gatepasses a good resultgateblocks a result under the bargateblocks a big drop but allows exactly 5 points
Stuck? One hint at a time
The solution unlocks after your first Check. Try first: that is where the learning happens.
main.py
Sign in to run your code.
Output
Press Run to see what your program prints.