Topic 6.2
Budgets and Guards
In one line
Guards stop one bug or one heavy user from emptying the budget: a cap on tokens per request, a daily limit per user, a global spend ceiling with an alert, and a kill switch you can flip without a deploy.
Think of it like this
A prepaid phone card. You can't spend more than the card holds, the phone warns you at 20 percent left, and a parent can block the card from their own phone if it is lost. Spending can't run away, because limits exist before the money is spent.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Guard
- A check that runs before a call and refuses it when a rule would be broken.
- Quota
- The amount one user or key may use in a period, such as 60 requests a day.
- Ceiling
- A hard global limit. When reached, new calls are refused.
- Alert threshold
- A lower level (such as 80 percent) at which people are notified while service still works.
- Kill switch
- A flag that turns a feature off or to a cheaper path immediately, without a deploy.
- Worst case
- The most one call could cost: the longest allowed input plus the longest allowed output.
Step by step
01Step 1: no guard, a runaway loop
A bug retries a failing request forever, or a script sends 100,000 requests. Nothing in the code stops it, and each call costs money. This is the situation guards are for.
02Step 2: bound one request
Set max_tokens and an input limit. Now a single call can't cost more than a known amount. Everything else counts calls against that known amount.
MAX_INPUT_CHARS = 8_000 # example limit: choose from your feature
MAX_OUTPUT_TOKENS = 400
def prepare(user_text):
if len(user_text) > MAX_INPUT_CHARS:
raise ValueError("message too long")
return {"max_tokens": MAX_OUTPUT_TOKENS, "text": user_text}03Step 3: count per user and per day
Keep a counter per user that resets when the day changes. In a real system this is a database row or a fast key-value store, because several servers share it. The example keeps it in memory and uses a simulated clock.
04Step 4: add the global ceiling and the alert
Track total spend in micro-dollars. Refuse when spend plus the worst case of the next call would pass the ceiling. Raise the alert once when 80 percent is spent. In the output you can see ana stopped by her own limit, bob stopped by the global ceiling, and the alert firing on the way.
05Step 5: reset on a new day, flip the switch
The last events show the counters resetting when the day number changes, and the kill switch refusing everything at once even for a user with room left. Both are one-line checks, and both are easy to get wrong without a test.
06Step 6: tell the user and keep a record
Each refusal returns a reason. Show the user a plain message, and write the reason to your logs and counters: a rising number of "global budget" refusals is an alert, not just a log line (Topic 11.4).
Try it yourself
- 1
Predict the refusals
Change
DAILY_REQUESTS_PER_USERto 2 andGLOBAL_BUDGETto 20_000. Which events are refused now, and does the alert still fire? Predict each line, then run. - 2
Order of checks
Swap the user-limit check and the global-budget check. Does any line change? Think about which refusal reason is more useful to show a user.
- 3
Add a second alert
Add an alert at 95 percent that fires once. What must you reset on a new day so it can fire again?
Code & diagrams
Spend is counted in micro-dollars (whole numbers). The clock is a counter in milliseconds, so the day changes when you decide.
Expected output
t=0 ana: ok (spent 2500)
t=1000 ana: ok (spent 5000)
t=2000 ana: ok (spent 7500)
t=3000 ana: refused (daily limit for this user) (spent 7500)
t=4000 bob: ok ALERT: 80 percent of today's budget is spent (spent 10000)
t=5000 bob: refused (global budget used up) (spent 10000)
t=6000 cy: refused (global budget used up) (spent 10000)
t=86400010 ana: ok (spent 2500)
t=86400020 ana: refused (kill switch is on) (spent 2500)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count the spend after the call
Change the global check to if self.spent > GLOBAL_BUDGET: (drop + EST_COST, in every language). The guard now compares only the money already spent, not the worst case of this call.
Myth vs fact
Myth
Rate limits from the provider protect my budget.
Fact
Provider limits protect the provider. A limit set high enough for your peak load can still let a runaway loop spend a lot. Your guards are yours to build.
Myth
An alert is as good as a limit.
Fact
An alert tells a person, who may be asleep. A limit stops the spend on its own. Use both.
Myth
Per-user limits are only for abuse.
Fact
They also protect honest users from one noisy neighbour, and they make cost per user predictable.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Count in a shared store with an atomic increment (a database row update or a Redis
INCR). A check-then-write across servers can let two requests both pass at the limit. - ▸
Reserve, then settle: reserve the worst-case cost before the call, then replace it with the real
usageafter. This keeps the ceiling safe under concurrency (Topic 6.10) without waiting for bills. - ▸
Different limits for different tiers (free, paid, internal) belong in configuration, with the kill switch able to target one feature or one tier.
- ▸
Practise the kill switch: flip it in a staging environment and check what users see. A switch nobody has tested is a hope.
Remember this
- 1
Why this exists. A model call costs money and the bill arrives later. A loop that calls the model forever, a user who pastes a book into every message, or a bot that is hammered by a script can spend a month of budget in an hour. Without guards you find out from the invoice. A guard is a rule that refuses a call before it is made.
- 2
Guard 1: bound each request. Set
max_tokenson every call and cap the length of user input (reject or truncate over a limit). Then the worst-case cost of one call is known: input cap plus output cap, priced (the formula from Topic 6.1). This is the base every other guard stands on (Topic 0.7 refuses a call whose worst case would pass the budget). - 3
Guard 2: a limit per user. A fair daily number of requests (or tokens, or dollars) per account stops one user using everything, and makes abuse visible. This site's AI tutor works this way: signed-in users only, with a daily limit per user (
AI_DAILY_LIMIT, default 60), counted in a database table. The lab on this site is limited similarly. - 4
Guard 3: a global ceiling and an alert. Add up spend across everyone and refuse new calls at a hard ceiling. Alert earlier (at 80 percent, say) so a person can act before users are refused. A ceiling protects the company; an alert protects the user experience.
- 5
Guard 4: a kill switch. One flag, read at call time, that turns the feature off (or onto a cheaper path) immediately, without a deploy. When something misbehaves at night, flipping a flag is faster and safer than shipping code. Test that it works before you need it.
- 6
Design the refusal. A refused call needs a plan: tell the user plainly ("you've reached today's limit, come back tomorrow"), degrade (use a cached or smaller-model answer, Topic 6.9), and log it. The example below keeps counters in whole micro-dollars, so there are no rounding surprises, and resets them when the simulated day changes.
Explain it without notes
Name four guards that protect an LLM feature's spend and what each one stops.
Why must the budget check happen before the call, using the worst case?
How does this site's AI tutor limit usage?
Why should a kill switch be a runtime flag and not a code change?
Practice
Write Quota(limit) with allow(user) that returns True and counts the call while the user has used fewer than limit calls, otherwise False. Test it with a limit of 2 and one user making 3 calls.
Add reset() to the quota that clears all counts, and show that a user who was refused is allowed after the reset.
Trade-offs
- ↔
Strict limits protect the budget but frustrate heavy honest users; tiers and clear messages soften this.
- ↔
Checking against the worst case refuses some calls that would have fit; a reserve-then-settle design reduces that waste.
- ↔
A kill switch adds a code path to test, but it is the cheapest incident tool you will own.
Done when you can
Done when every call has
max_tokensand an input cap.Done when you can write a per-user daily limit that resets with the day.
Done when you can explain the global ceiling, the alert and why both exist.
Done when you can show a kill switch flipped at runtime.
Done when refusals tell the user why and are counted in your metrics.