Command Palette

Search for a command to run...

Hectal
Phase 9: Evaluation and Testing
LLM Engineering · Exercise 4 of 6 PreviousNext
Advanced

Write the CI gate

Write gate(results, threshold, critical). results is a dict from case id to True or False, threshold is the minimum pass rate, and critical is a list of case ids that must pass.

Return (status, problems). problems is a list of readable strings: one if the pass rate is below the threshold (include the word rate), and one per failed critical case (include its id). status is 1 when there are any problems, otherwise 0. If results is empty, that is a problem too (no cases).

What Check tests

  1. everything passes
  2. a rate below the threshold blocks
  3. a failed critical case blocks even with a high rate
  4. a critical case that was not run counts as failed
  5. no cases at all is a problem, not a crash

Stuck? One hint at a time

The solution unlocks after your first Check. Try first: that is where the learning happens.

main.py
Sign in to run your code.

Output

Press Run to see what your program prints.