Topic 7.7
Dangerous Tools
In one line
Tools that change the world (refund, delete, send, deploy) turn a model mistake or a hostile instruction into a real incident. Separate read from write, ask a human before side effects, give each tool the least power it needs, make writes safe to repeat, and keep an audit log.
Think of it like this
A company credit card. Anyone may look at the statement. Small purchases need a manager's signature, large ones two signatures, and the card cannot buy anything on a banned list. Every purchase is written in a ledger. Nobody assumes the cardholder is dishonest. They assume mistakes and theft happen, and they limit the damage when they do.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Side effect
- A change a tool makes outside itself: a sent email, a changed record, a deleted file.
- Least privilege
- Giving a tool or a user only the permissions needed for the job and nothing more.
- Confirmation
- Asking a person to approve an action, showing the exact details, before it runs.
- Idempotency key
- A unique id for an intended action. A repeated key means 'already done', not 'do it again'.
- Audit log
- A record of every tool call and decision, kept so you can reconstruct what happened.
- Blast radius
- How much damage one mistake or attack can cause.
Step by step
01Step 1: label the risk
Each registry entry carries read or write next to its function. search_orders is a read. refund_order is a write. The label drives the policy.
02Step 2: ask before a write
For refund_order the runner prints the question with the exact arguments and waits for a decision. The decisions here are scripted (yes, then no); in an app they come from a button the user presses. A decline becomes the tool message the user declined this action.
03Step 3: refuse what is not registered
export_all_customers is not in the registry, so it cannot run, whatever the model or a document says. The audit log still records the attempt: repeated attempts like this are a signal worth alerting on.
04Step 4: limits depend on the user
The second example looks up the role in the session. A support agent can refund up to 50, a manager up to 500, and only a manager may use delete_account. The model never supplies the role.
05Step 5: safe to repeat
The second call in that example reuses key k1 and returns 'already done'. A retry or a looping model cannot refund twice.
Try it yourself
- 1
Change the decisions
In the first example change the scripted decisions to
[False, True]. Predict which refund goes through and what the audit log says. - 2
Raise a limit
In the second example change the support limit to 100. Which call changes? Why is this limit better stored in configuration, with changes reviewed, than chosen by the model?
- 3
Add a tool
Add
send_emailto the first example as awritetool. What would the confirmation question need to show so the human can judge it (recipient, subject, body)?
Code & diagrams
The human's answers are scripted (yes, then no). In an app they come from a confirmation dialog.
Expected output
call_1 search_orders: {"orders":["A17"]}
asking the user: allow refund_order {"order_id":"A17","amount":20}?
call_2 refund_order: {"refunded":20,"order_id":"A17"}
asking the user: allow refund_order {"order_id":"A18","amount":15}?
call_3 refund_order: {"error":"the user declined this action"}
call_4 export_all_customers: {"error":"unknown tool"}
audit log:
call_1 search_orders read, ran without asking
call_2 refund_order approved by user
call_3 refund_order declined by user
call_4 export_all_customers refused (not registered)The role comes from the session, never from arguments the model wrote.
Expected output
1 support refund_order: refunded 20 on A17
2 support refund_order: already done (same idempotency key)
3 support refund_order: refused: 80 is above the limit 50 for support
4 manager refund_order: refunded 80 on A18
5 support delete_account: refused: support may not use delete_accountBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Approve everything automatically
In the first example, delete the if not next(decisions) block so every write is approved (keep the approved by user line), and run the Python version.
Myth vs fact
Myth
A good system prompt will stop the model from doing harmful things.
Fact
Prompts guide behaviour and can be overridden by later text. Real protection is in code: permissions, confirmations, limits.
Myth
Confirmation prompts make an agent safe.
Fact
Only if they show the true action and people read them. Constant prompts get ignored, so keep them for what matters and combine them with limits.
Myth
Logging is only for debugging.
Fact
Audit logs are how you detect misuse, prove what happened and notice a model starting to behave differently.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Separate the planning from the doing: let the model propose an action as data (a draft email, a refund request), show it, and have ordinary code execute it after approval. The model never holds the credentials.
- ▸
Credentials: give each tool its own narrow credential or token and keep them outside the prompt. A leaked prompt or log must not leak a key (Topic 2.4).
- ▸
Rate-limit writes per user and per request, alert on denied or refused calls, and review the log for tools that are called but never approved.
Remember this
- 1
Why this exists. A read-only tool can at worst show the wrong data. A write tool (send an email, refund an order, delete a file, run a command, deploy) can do harm that cannot be undone. The model may pick the wrong tool, misread a request, or be steered by hostile text in a web page, an email or a document it was asked to summarise (prompt injection, Phase 10). You cannot make the model perfectly reliable. You can make sure that a mistake cannot do much.
- 2
Read versus write. Label every tool by its risk: read (no side effects), write (changes something, reversible), destructive (cannot be undone) or external (sends something to other people). Reads can run automatically. Writes need a policy. Destructive and external actions usually need a human yes, or should not be tools at all. The first example keeps the label next to each function in the registry.
- 3
Confirmation. Before a write runs, show the human exactly what will happen in plain words and the exact arguments (the model's summary is not enough, it can differ from the call), and run it only on an explicit yes. A decline goes back to the model as a tool message so it can explain or offer another way. Protocol guidance for tools also says there should always be a human in the loop with the ability to deny calls (Topic 7.8). Too many prompts train people to click yes; reserve them for actions that matter.
- 4
Least privilege. Give the tool only the power it needs. A report tool gets a read-only database login, not the admin one. A file tool gets one folder. A refund tool has a maximum amount that depends on who is using it (the second example: support up to 50, manager up to 500). Take the user and role from your authenticated session, never from arguments the model wrote (recall from Topic 7.4: authority comes from the session, not the arguments). A tool that does not exist cannot be misused, so do not build a generic 'run any SQL' or 'run any command' tool for a model.
- 5
Make writes safe to repeat. Loops and retries mean a call may run twice. Give every write an idempotency key (a unique id for the intended action) and have the tool remember keys it has done: a repeated key returns 'already done' instead of refunding twice. Prefer reversible designs (move to trash, mark as cancelled, create a draft) over irreversible ones (delete, send).
- 6
Audit everything. Write one log line per tool call: who, which tool, the arguments (with secrets removed), the decision (ran, declined, refused) and the result status. When something goes wrong you need to answer 'what did it do and why' from the log. Add limits too: a maximum number of write calls per request and per user per day, so a looping or hijacked model cannot do unlimited damage. Phase 10 covers prompt injection and Phase 11 covers observability.
Explain it without notes
Why do read tools and write tools need different treatment in a tool runner?
What is least privilege for a tool, with an example?
What is an idempotency key and why do tool loops need it?
Practice
Write needs_confirmation(risk) that returns True for "write" and "destructive" and False for "read". Make unknown risks need confirmation (fail closed).
Write audit_line(call_id, tool, decision) that returns "<call_id> <tool> <decision>" and Audit with add(line) and count(decision) that counts lines ending in the decision.
Trade-offs
- ↔
Confirmations make writes safe and slow the workflow; use them for actions that matter and automate the rest.
- ↔
Narrow tools are safer and need more tools; a generic tool is flexible and is a large risk.
- ↔
Audit logs cost storage and care with private data and are what makes incidents explainable.
Done when you can
Done when every tool is labelled read or write (or destructive) in the registry.
Done when writes need an explicit approval that shows the exact arguments.
Done when roles, limits and credentials come from the session and configuration, not the model.
Done when writes carry idempotency keys.
Done when every call and decision lands in an audit log.