Topic 12.7
Prompts and Models as Configuration
In one line
Prompts, model names, limits and thresholds change much more often than code. Keep them in versioned configuration, switch features with flags, roll changes out to a percentage of users using a stable hash, watch the metrics, and roll back by changing one value instead of redeploying.
Think of it like this
A restaurant's menu board and its kitchen. Changing today's special means changing the board, not rebuilding the kitchen. And a new dish is first offered to a few tables to see whether people like it, and removed quickly if it makes them ill. Configuration is the menu board: values you can change without changing the machinery. A rollout is the trial with a few tables.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Configuration
- Values that control behaviour and can change without changing code.
- Feature flag
- A named switch that turns a behaviour on or off at run time.
- Rollout
- Turning a change on for a growing share of users.
- Bucket
- A stable number from 0 to 99 computed from a user id, used to decide who is in a rollout.
- Hash
- A function that turns text into a number, always the same number for the same text.
- Kill switch
- A flag whose job is to turn something off fast.
- Rollback
- Going back to the previous behaviour.
Step by step
01Step 1: the prompt in the code
Initially the prompt is a string in a function. To change one word you edit code, review, build and deploy. Meanwhile you cannot try the new wording on a few users and compare.
02Step 2: move the values out
The example has a CONFIG dictionary with the model, the current prompt version, and a rollout percentage for v2. In a real service this is a validated file or a config service entry.
{
"model": "small",
"prompt_version": "v1",
"rollout": { "prompt_v2": 30 }
}03Step 3: a stable bucket
bucket turns a user id into 0 to 99 with a hash written by hand: multiply the running value by 31, add the character code, keep it small with a remainder. A last step multiplies by 7919 and takes remainders again, which spreads neighbouring ids such as u1 and u2 far apart. It is the same in all three languages, so the same user always lands in the same bucket. Real systems use a good library hash (such as murmur) for better spread, and include the flag name so different experiments choose different users.
04Step 4: roll out and read the result
With 30 percent, the users with a bucket below 30 get v2. Raising to 60 percent keeps those users and adds more: nobody flips back. The line of versions shows exactly who has which.
05Step 5: roll back
Set the percentage to 0 and everyone is back on v1 without a deploy. Because rollout and rollback are the same operation (changing one number), you can practise it.
06Step 6: tie it to measurement
A rollout is only safe if you can see the difference between the groups. Log the version on every request, compare error rate, cost, latency and feedback by version, and pre-decide the numbers that mean "stop". Use the eval gate of Topic 9.5 before the first percent.
Try it yourself
- 1
Find your bucket
Add your own name to the user list and see its bucket. At what rollout percentage would you get the new prompt?
- 2
Same users, higher percentage
Compare the 30 percent and 60 percent lines. Check that every user on
v2at 30 percent is also onv2at 60 percent. - 3
Two experiments
Make the hash include the flag name (
flag + ":" + user_id). Run it for two flags with the same percentage. Do they pick the same users?
Code & diagrams
The same user always gets the same bucket, so a user never flips between versions during the rollout.
Expected output
buckets: u1=57 u2=76 u3=95 u4=14 u5=33 u6=52 u7=71 u8=90 u9=9 u10=70
rollout 30 percent: 2 of 10 users on v2 -> v1 v1 v1 v2 v1 v1 v1 v1 v2 v1
rollout 60 percent: 5 of 10 users on v2 -> v2 v1 v1 v2 v2 v2 v1 v1 v2 v1
rolled back: 0 of 10 users on v2 -> v1 v1 v1 v1 v1 v1 v1 v1 v1 v1Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Bucket by something that is not stable
Replace the hash with len(user_id) % 100 (JavaScript: userId.length % 100, Java: userId.length() % 100).
Myth vs fact
Myth
Prompts are code, so they must go through the normal deploy.
Fact
They need versioning and review like code, but a config path with evals and a rollout gives faster and safer change than a full deploy.
Myth
A random number per request is a fine way to do a 10 percent rollout.
Fact
Then one user sees both versions, results are noisy and conversations change style midway. Hash a stable id.
Myth
Flags are free to keep around.
Fact
Old flags are dead code paths and test combinations. Remove a flag after it reaches 100 percent.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cache the config in each running copy and refresh it on an interval or by a signal. Decide what happens when the config service is down: keep the last good config, never fall back to an empty one.
- ▸
Validate on write, not just on read: a bad value should be rejected when someone saves it, with a diff and a reviewer, not discovered by the first user.
- ▸
Rollouts by tenant (all users of a pilot customer) and by percentage can be combined: allow-list pilot tenants first, then percentages.
- ▸
Hosted feature-flag services and open-source ones exist; the hash-bucket idea is the same inside them, and you can start with a file.
Remember this
- 1
Why this exists. In an LLM product the things that change most are not code: the wording of a prompt, the model name, the temperature, the maximum tokens, the quota of a plan, the threshold of a router (Topic 12.6). If each change needs a code change, a review and a deploy, you change them rarely and carelessly. If they are loose strings someone edits in production with no history, you cannot tell what changed when quality dropped.
- 2
What belongs in configuration. Prompt templates and their versions (Topic 3.8), model names per feature, sampling settings, limits and quotas, the router thresholds, price tables, and feature flags. Keep secrets out: keys belong in a secrets store or environment variables (Topic 2.4), not in config files in the repository.
- 3
Make config safe. Store it as a versioned file in the repository, or in a config service, with a schema that is validated at load time (Topic 4.5): an unknown model name or a negative limit fails at startup or at the moment of the change, not in the middle of a user's request. Every change has an author, a time and a way back. Load it once per process and refresh on a signal, not by reading a file on every request.
- 4
Feature flags. A flag is a named switch that decides at run time which code path or which config a request uses. Flags let you merge and deploy unfinished work turned off, switch a feature on for one tenant, and turn something off in seconds when it misbehaves (a kill switch, useful for a tool that started causing trouble or a model that started failing).
- 5
Percentage rollout. To try a new prompt on 10 percent of users, you must pick the same 10 percent every time, or one user would see the old and new prompt alternately. Compute a bucket from a stable id: hash the user id (and the flag name) to a number from 0 to 99, and enable the flag if the bucket is below the percentage. The same user always gets the same answer, on every machine. Raise the percentage in steps (5, 25, 50, 100) while watching quality, errors, cost and latency per version (Phase 11).
- 6
Roll back by changing a value. Because the percentage is data, rollback is setting it to 0, which takes effect without a deploy. Plan the rollback before the rollout: know which metric or eval score means "go back", who can change it, and how fast the change reaches all running copies.
- 7
Trace the version. Log the prompt version, model and flag values with every request and store the prompt version with each saved reply (Topic 12.3). When quality drops, the first question is "what changed?", and the logs should answer it. Run the evals (Phase 9) on a prompt change before any rollout, and compare versions online with an A/B test (Topic 9.7).
Explain it without notes
Why treat prompts and model names as configuration?
How does a percentage rollout choose its users, and why must it be stable?
What makes a rollback fast and safe?
What should be logged so that a quality drop can be traced to a change?
Practice
Write bucket(text) with the same hash as the example and print the buckets of "anna" and "ben".
Write enabled(user, flag, percent) using bucket(flag + ":" + user) < percent and show that percent 0 enables nobody and 100 enables everybody for ten users.
Trade-offs
- ↔
Config changes are fast and need discipline: without review and validation, fast also means fast to break.
- ↔
More flags give more control and more combinations to test and clean up.
- ↔
Gradual rollouts take longer to finish and catch problems on 5 percent of users instead of all of them.
Done when you can
Done when you can list what belongs in configuration and what must not.
Done when you can explain a hash bucket rollout and why it is stable.
Done when you can roll back by changing one value.
Done when you can name what to log to trace a regression to a change.
Done when you can say when to remove a flag.