PHASE 9Advanced ~23 min· topic 5 of 8

Topic 9.5

Least Privilege for AI Features

In one line

Least privilege means each part of the system gets only the power it needs. For an AI feature that means: tools are built per request for the signed-in user, the server (not the prompt) decides who may do what, risky actions need approval, and the model's arguments are never trusted.

Think of it like this

A hotel key card. The card you get opens your own room, the lift and the gym, and nothing else. If someone steals it or tricks the porter into using it, they can reach only those three doors. The card does not rely on the thief being polite. Its power is small by design.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Least privilege: Each component gets the smallest set of powers it needs to do its job.
Excessive agency: A model that has more tools, permissions or freedom than the task needs (OWASP LLM06).
Session: The server's record of who is signed in, found from a cookie or token on each request.
Closure: A function that remembers the variables around where it was created. toolsFor(session) uses one to hold the user.
Authorisation: Deciding whether this user may do this thing. It is different from authentication, which is proving who they are.
Read-only tool: A tool that looks something up and changes nothing.
Scope: The set of data or actions a tool or token can reach.
Rate limit: A cap on how many times something may happen in a time window.

Step by step

01Build tools per request

The first example has a small order table with two owners. toolsFor(user) returns a getOrder tool whose execute compares the order's owner with user, a value from the session. The tool's input schema is only { id }.

The scripted model (a stand-in, in a real app it is your provider's model) is fooled: it asks for ORD-2001, which belongs to someone else, and also tries a tool named cancelOrder that was never given to it.

02Read what happened from the stream

The example reads the SDK's fullStream and prints each tool result and tool error as it arrives. The foreign order gives { found: false }, identical to a missing order. The unknown tool becomes an error naming the tools that do exist, and nothing runs.

03Drop fields the model should not choose

The model adds userId: "ravi" to its arguments, hoping to act as someone else. Because the schema does not have that field, validation removes it before execute sees it, and the tool uses the session user anyway. The second example checks this with the model's raw input printed next to what the tool received.

Drop fields the model should not choosediagram

Rendering diagram…

04Cap the number of calls

A loop that keeps calling a tool costs money and may probe your data. The second example adds a counter per user and tool; the third call in one request is refused with rate-limited. In a real app the counter lives in a shared store such as Redis, keyed by user and minute.

05Keep write tools out of reach

Mitra's lookup tool can read. The cancel tool from Topic 8.9 should be offered only where cancelling is the task, and it always asks for approval first. The summary mode of Topic 9.2 has no tools at all. The table of modes and tools is short enough to review in one glance, which is the point.

Try it yourself

1
Remove the owner check
In the first example, change the condition if (!order || order.owner !== user) to if (!order) and run it.
The first tool result now contains Ravi's item. The model was fooled the same way as before; only the missing check changed the outcome.
2
Let the model choose the user
In the second example, change the schema to z.object({ id: z.string(), userId: z.string() }) and run it.
The tool now receives userId: "ravi". A field you offer is a field an attacker can set. Notice too that the second and third calls never reach the tool: their input has no userId, so validation rejects them and the rate limit never fires.

Code & diagrams

A fooled model asks for someone else's order

The model is a stand-in that always asks for the wrong things. The defence is not in the prompt: the tool checks the owner, and the tool the model invented was never given to it.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

tool result: {"found":false}
tool error: AI_NoSuchToolError: Model tried to call unavailable tool 'cancelOrder'. Available tools: getOrd
text: "I"
text: " could"
text: " not"
text: " find"
text: " that"
text: " order."
audit log: [{"user":"asha","tool":"getOrder","id":"ORD-2001","result":"denied-or-missing"}]

Extra fields and too many calls

The scripted model sent `userId: "ravi"` with its first call. The SDK checks the input against the schema when the call arrives, so the field is already gone in the `model asked` line, and the tool never sees it. The third call hits the per-user limit even though each single call looked fine.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

model asked: {"id":"ORD-1"}
tool returned: {"found":false}
tool received: {"id":"ORD-1"}
model asked: {"id":"ORD-2"}
tool returned: {"found":false}
model asked: {"id":"ORD-3"}
tool returned: {"error":"rate-limited"}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust the owner the model names

In the second example, change the schema to z.object({ id: z.string(), userId: z.string() }), then run it.

terminal

$ Console, the first call

── what you'll see ──

model asked: {"id":"ORD-1","userId":"ravi"}

tool returned: {"found":false}

tool received: {"id":"ORD-1","userId":"ravi"}

(the next two calls print "model asked" only: validation rejects them)

Myth vs fact

Myth

Our system prompt says to only use the signed-in user's data, so tools are safe.

Fact

A prompt is advice to a model that can be fooled. Authorisation must be code that runs in the tool with the session user.

Myth

Telling the user "order not found" for someone else's order is unhelpful.

Fact

A different message for "exists but not yours" lets an attacker discover which ids exist. Use the same answer for both.

Myth

Approval screens are enough, the tool can be powerful.

Fact

People click approve. Keep the tool narrow too, so an approved mistake has a small effect.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Prefer tokens and database roles that match the user. If your tool talks to a database or an internal API, call it with the user's own credential or a row-level security policy (a database rule that limits which rows a role can see), so a bug in your tool code still cannot read other users' rows.
▸
Separate the planning step from the acting step when it fits: one model call proposes a change as plain data, your code validates it against rules, and only then does a separate function perform it. The model never holds the power; it holds a suggestion.
▸
Review tool lists like you review database permissions. A short table of mode, tool, read or write, approval needed, and rate limit goes into code review whenever a tool is added.

Remember this

1
The problem this solves: Topic 9.2 showed that the model can be fooled. If a fooled model holds a master key (a tool that can read any order, or cancel any order), the attacker holds it too. Least privilege makes the master key not exist. This is the defence against LLM06, Excessive Agency in the OWASP Top 10 for LLM Applications (2025): a model given more functions, permissions or freedom than the task needs.
2
Rule 1: the prompt is not a permission system. "Only show the user their own orders" written in the system prompt is a request, not a lock. Permission checks belong in your server code, in the function that runs when a tool is called, using the identity from the signed-in session (the server's record of who is logged in), never an identity the model or the browser supplies.
3
Rule 2: build the tools per request, for this user. Make a function toolsFor(session) that returns the tools. The user's identity is captured in the closure, so a tool cannot even express "for user X". The model's input schema contains only what the model must choose (an order id), never a userId. If the model adds an extra field anyway, validation drops it.
4
Rule 3: ownership checks live in the tool. The tool looks up the order and compares its owner with the session user. A missing order and someone else's order give the same answer (found: false). If they differed, the model, and through it an attacker, could probe which order ids exist.
5
Rule 4: give each mode only the tools it needs. A summary mode has no tools. A help mode has read-only lookups. Only a mode that must act has write tools (like the cancel tool in Topic 8.9), and each write goes through approval (Topic 8.6). A tool the model was never given cannot be called, and the SDK reports an attempt as an error instead of running anything.
6
Rule 5: the model's arguments are untrusted input. Validate them like form fields: types, ranges, allowed values. A refund tool takes an order id, not an amount the model computed. Where an amount is needed, the server calculates it from the stored order. Add rate limits per user and per tool, so a looping or fooled model cannot do a thousand lookups.
7
Rule 6: show the user what the model did. Phase 8 gave you tool parts. Show each tool call with its input, so a person can see "looked up ORD-2001" and notice something odd. Visible actions turn a silent compromise into a noticeable one.
8
Detect and verify. Detect: log every tool call with user id, tool name and result status, and alert on denied lookups (found: false for ids that belong to others). Verify: a test signs in as user A, forces the model (the scripted stand-in) to ask for user B's order, and asserts that nothing from B's order appears and that the denial was logged. The live examples below do exactly this.

Explain it without notes

Why is "only use the current user's data" in the system prompt not an access control?

How does building tools per request with toolsFor(session) reduce the risk?

Why should "not yours" and "does not exist" look the same to the model?

What does OWASP mean by excessive agency, and how do you limit it?

Practice

Add a read-only listOrders tool to the first example that returns the ids of the signed-in user's orders only. Which line makes it safe?

Change the second example so the limit is per tool per request and is returned as { error: "rate-limited", retryAfterSeconds: 60 }. What should the UI tell the user when it sees that?

Trade-offs

↔
Narrow tools make Mitra less flexible: it cannot answer questions that need a tool you did not give it. That is deliberate; add a tool when a real task needs it, with its own checks.
↔
Per-request tools and per-user credentials add plumbing. The saving is that a bug in prompt wording never becomes a data breach.

Done when you can

I can explain why permissions belong in server code and not in the prompt.
I can write toolsFor(session) so the identity is not a model argument.
I can make "not yours" and "missing" return the same answer and still log the difference.
I can give each mode only the tools it needs.
I can test a fooled model against my tools and assert the denial.

Command Palette

Words you'll meet

Step by step

01Build tools per request

02Read what happened from the stream

03Drop fields the model should not choose

04Cap the number of calls

05Keep write tools out of reach

Try it yourself

Remove the owner check

Let the model choose the user

Code & diagrams

Break it on purpose

Trust the owner the model names

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can