Command Palette

Search for a command to run...

Hectal
Phase 9: Evaluation and Testing
LLM Engineering · Exercise 3 of 6 PreviousNext
Intermediate

Diff two eval runs

Write compare(old, new). old and new are dicts from case id to True (passed) or False (failed), with the same ids.

Return a dict with three lists of ids, in the order of old: "fixed" (failed before, passes now), "broken" (passed before, fails now) and "still_failing" (failed both times).

What Check tests

  1. the example from the task
  2. nothing changed
  3. the order of old is kept
  4. a passing case that stays passing is not listed

Stuck? One hint at a time

The solution unlocks after your first Check. Try first: that is where the learning happens.

main.py
Sign in to run your code.

Output

Press Run to see what your program prints.