Versions, a regression set and a gate
A fake classifier reads the prompt text. model-b stands for a provider update that changes the reply format.
Store prompts in files with an id, a version and a changelog, and run a small regression set before every change. A change, or a model update, that lowers the score is caught before users see it.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write pass_count(cases, fn) that returns how many (input, expected) cases the function fn(input) answers correctly.
Write can_ship(old_score, new_score) that returns True only if the new score is at least the old one, and use it to gate the scores 3 -> 4 and 4 -> 2.
Explain it without notes
What should a stored prompt record contain, and why?
How does a regression set protect you, and how can it mislead you?
Why can a prompt that worked last month fail today, and what do you do about it?
Expected output
ticket-label v1 on model-a: 0/4 baseline (first draft)
ticket-label v2 on model-a: 3/4 ok (ask for the label only)
ticket-label v3 on model-a: 4/4 ok (refund requests are billing)
ticket-label v4 on model-a: 0/4 BLOCKED (best so far 4/4) (shorter wording)
ticket-label v3 on model-b: 0/4 after a model update