Topic 3.8
Versioning and Testing Prompts
In one line
Store prompts in files with an id, a version and a changelog, and run a small regression set before every change. A change, or a model update, that lowers the score is caught before users see it.
Think of it like this
A restaurant recipe card. When the chef changes the salt, the card gets a new date and a note ("less salt, guests complained"), and someone tastes the dish before it goes on the menu. If the supplier changes the flour, the dish is tasted again. A prompt is the recipe, and the test set is the tasting.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Prompt id
- A stable name for a prompt, such as ticket-label.
- Version
- A number that goes up each time the prompt text changes.
- Changelog
- A one-line note saying why a version was made.
- Regression set
- Test inputs with expected answers, used to check that a change did not make things worse.
- Gate
- A rule that blocks a change if its score falls.
- Drift
- A change in behaviour over time, for example after a model update.
- Pinning
- Using an exact model version instead of a moving alias such as latest.
Step by step
01Step 1: prompts become records
Each prompt version is a record with id, version, text and changelog. In a real project these are files such as prompts/ticket-label.v3.txt, committed to git, so the history is free.
PROMPT = {"id": "ticket-label", "version": 3, "text": "Reply with only one of: billing, shipping, other.", "changelog": "refund requests are billing"}02Step 2: a regression set
Four tickets with the label you expect: charged twice, parcel not delivered, refund, change email. They are not the prompt's examples. They only measure.
03Step 3: score every version
Run each version through the fake model and count exact matches. Version 1 ("Label the ticket.") gets a chatty reply and scores 0 of 4. Version 2 asks for the label only and scores 3 of 4. Version 3 adds the refund rule and scores 4 of 4.
04Step 4: the gate blocks a regression
Version 4 reads "Reply in one word" and drops the phrase the fake model needs, so it scores 0 and is blocked. A real model may handle the rewording better or worse. The test tells you which.
05Step 5: a model update changes the result
The last line runs version 3 on model-b, which adds a "Label: " prefix. Zero passes, with no prompt change at all. The same regression set that guards your edits also guards your model upgrades.
Try it yourself
- 1
Add a failing case
Add the case
("my invoice is wrong", "billing")toCASES. Which versions fail it, and what change to the prompt would the fake model need? (Hint: the fake model has no rule forinvoice.) In real life you would improve the prompt and add the case to the set so it never regresses. - 2
Make a gate that keeps the best
Look at how
bestis updated. Change the code so a blocked version does not replace the best score (it already does not), and then add a version 5 that scores 4 of 4. Is it "ok"? - 3
Pin the model
Add a
modelfield to each prompt record and run each version against the model it was written for. Where would you store which model is in production?
Code & diagrams
A fake classifier reads the prompt text. model-b stands for a provider update that changes the reply format.
Expected output
ticket-label v1 on model-a: 0/4 baseline (first draft)
ticket-label v2 on model-a: 3/4 ok (ask for the label only)
ticket-label v3 on model-a: 4/4 ok (refund requests are billing)
ticket-label v4 on model-a: 0/4 BLOCKED (best so far 4/4) (shorter wording)
ticket-label v3 on model-b: 0/4 after a model updateBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Test on the examples
Make CASES contain only the first two tickets (the ones the prompt is easiest on) and run.
Myth vs fact
Myth
If a prompt worked last month it still works.
Fact
The prompt text is the same but the model may have changed. Pin versions and re-run the set when anything changes.
Myth
Prompts are text, so git is enough.
Fact
Git records what changed, not whether it is better. You also need a score on a test set.
Myth
Passing the regression set means the prompt is good.
Fact
It means it did not get worse on those cases. A small set can hide gaps. Grow it from real failures and move to full evaluation in Phase 9.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the prompt id and version with every model call (and the model version). When quality dips, you can filter logs by version and compare. Without that field you are guessing (Phase 11).
- ▸
Run the regression set in CI on every change to a prompt file, using a recorded or fake model for speed and a real model nightly. Cost per run is small: 30 cases at 300 tokens each is 9,000 tokens.
- ▸
Roll out prompt changes like code: shadow-run the new version on live traffic, compare, then switch a percentage of users (Phase 12). A prompt is a deployable artifact.
Remember this
- 1
Why this exists. Prompts change constantly: someone fixes one wrong answer, and three others quietly break. Without a record you cannot answer "what changed, when, and why?". Without a test you only find out from angry users. The same is true when the model behind the prompt changes, because a prompt tuned for one model can behave differently on its successor.
- 2
Prompts in files with ids and versions. Keep each prompt in a file or table row with an id (
ticket-label), a version (3), the text and a changelog line saying why it changed. The application asks for "ticket-label, version 3", not for a string in code. Rolling back is then one setting, and logs can record which version produced each answer. - 3
A small regression set. A regression set is a list of inputs with the answer you expect, kept separate from the prompt's examples (Topic 3.3). Start with 10 to 30 cases: common ones, the awkward ones that once failed, and a few edge cases. Run every prompt version on it and compute a score. The example uses four tickets and exact-match checks, which suit classification.
- 4
A gate. A change may ship only if its score is at least the best score so far. In the example, version 4 "shorter wording" scores worse and is blocked, even though it looks like a harmless edit. This is the same idea as unit tests in CI (see the CI/CD course), applied to prompts.
- 5
Why prompts drift when models update. Providers retire models and release updates. A new version may add chatty prefixes, change formatting, or read an instruction differently. The example's
model-banswers "Label: billing" instead of "billing": same prompt, different model, zero passes. Pin an exact model version where the provider allows it, and re-run the regression set on every model change before switching. - 6
Keep it honest. A small set can mislead: passing 4 of 4 is not proof of quality, and fixing the set's failures by editing the prompt can overfit to those cases. Grow the set from real production failures, hold some cases back, and move to proper evaluation (Phase 9) as the product matures. Fake models in tests make the run free and repeatable; run the real model on the same set regularly too.
Explain it without notes
What should a stored prompt record contain, and why?
How does a regression set protect you, and how can it mislead you?
Why can a prompt that worked last month fail today, and what do you do about it?
Practice
Write pass_count(cases, fn) that returns how many (input, expected) cases the function fn(input) answers correctly.
Write can_ship(old_score, new_score) that returns True only if the new score is at least the old one, and use it to gate the scores 3 -> 4 and 4 -> 2.
Trade-offs
- ↔
A bigger regression set catches more but takes longer and costs more per run; keep a fast core set and a larger nightly set.
- ↔
Strict exact-match checks are simple but harsh on free text; use rules (word limits, required phrases) or a judge model for those (Phase 9).
- ↔
Pinning a model avoids surprises but means you must plan migrations when it is retired.
Done when you can
Done when you can store a prompt with id, version and changelog.
Done when you can build a regression set that is separate from the prompt's examples.
Done when you can gate a prompt change on its score.
Done when you can explain model drift and how pinning and re-testing handle it.