PHASE 7Intermediate ~24 min· topic 9 of 9

Topic 7.9

Playbook: A Safe Tool Runner

In one line

One small class that does what this phase taught: a registry, schemas, argument validation, confirmation for writes, a timeout, errors fed back, an audit log, and a step-limited loop around it. Copy it, then plug the real model in.

Think of it like this

The front desk of a secure office building. Visitors (the model's requests) are checked against a list, asked for ID, sent to the right floor, and sometimes need an employee to come down and sign for them. Everything is written in the visitor book. One desk does all of this, so every visitor gets the same treatment and nobody has to remember the rules.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Registry: Your table of allowed tools: name, description, schema, function, risk and timeout.
Runner: The code that takes a tool call and applies every check before and after running the function.
Outcome: How a call ended: ok, refused, invalid, declined, failed or timeout.
Audit log: One line per call with its outcome, kept for later investigation.
Scripted model: A fake model that returns prepared messages, so the loop can be tested without a network.
specs(): The method that builds the tools list for the request from the registry.

Step by step

01Step 1: register the tools

Three tools: get_order (read), refund_order (write, needs a human yes) and slow_report (read, but its simulated time of 5000 ms is over its 1000 ms deadline). specs() returns the list the model would see.

02Step 2: one entry point for every call

run(call) does the checks in a fixed order: registered, JSON, object, schema, confirmation, run, deadline. Each early exit returns a tool message and an audit line.

Step 2: one entry point for every calldiagram

Rendering diagram…

03Step 3: wrap it in a loop

run_agent repeats: take the model's reply, return its text if there are no tool calls, otherwise run each call through the runner. It has a step limit and returns a clear message when it hits it.

04Step 4: three scenarios

Run 1: the model sends order_id as a number, the runner replies expected string, the model fixes it and then a refund is approved. Run 2: the user declines. Run 3: the model asks for a slow report forever and the limit stops it.

05Step 5: read the audit log

The last line lists every call with its outcome: c1 get_order invalid, c2 get_order ok, c3 refund_order ok, c4 refund_order declined, c5 slow_report timeout and so on. That one line explains the whole session.

06Step 6: plug in the real model

Replace the scripted reply with a real API call (see the shown code), pass runner.specs() as tools, and let runner.run(call) produce the tool messages. Test with the lab's models first.

Try it yourself

1
Predict the audit log
Before running, write down the audit log you expect for the three runs. Then run and compare, line by line.
2
Add a tool
Register cancel_order as a write tool with one string argument. Which parts of the runner did you have to change? (None: that is the point.) Add a scripted call that gets declined.
3
Tighten a deadline
Give get_order a 50 ms deadline (its simulated time is 80 ms) and rerun. Which outcome appears in the audit log, and which message would the model see?

Code & diagrams

The safe tool runner and its loop New tab

The timeout is simulated with numbers. Real code enforces the deadline while waiting (Topic 7.6). In Java the course's Json class stands in for Jackson.

Expected output

tools sent to the model: get_order, refund_order, slow_report
run 1: a typo, then a refund the user approves
  step 1: get_order -> {"error":"order_id: expected string"}
  step 2: get_order -> {"order_id":"A17","status":"shipped","total":40}
  step 3: refund_order -> {"refunded":20,"order_id":"A17"}
  final: Refunded 20 on order A17.
run 2: the user declines
  step 1: refund_order -> {"error":"the user declined this action"}
  final: Understood, I did not refund order A18.
run 3: a model that never stops
  step 1: slow_report -> {"error":"timed out after 1000 ms"}
  step 2: slow_report -> {"error":"timed out after 1000 ms"}
  step 3: slow_report -> {"error":"timed out after 1000 ms"}
  final: stopped: step limit 3 reached
audit log: c1 get_order invalid; c2 get_order ok; c3 refund_order ok; c4 refund_order declined; c5 slow_report timeout; c6 slow_report timeout; c7 slow_report timeout

Plugging in a real modelpython

Not run here: it needs a network call and your own GROQ_API_KEY (from an environment variable, never in code). Python and JavaScript use the OpenAI SDK pointed at Groq's OpenAI-compatible endpoint. Java uses plain HTTP, which does not depend on an SDK version.

import json
import os

from openai import OpenAI  # pip install openai. Works with any OpenAI-compatible provider through base_url.

client = OpenAI(api_key=os.environ["GROQ_API_KEY"], base_url="https://api.groq.com/openai/v1")


def real_model(messages, tools):
    """Same job as the fake model's script: return the next assistant message as a dict."""
    response = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=messages,
        tools=tools,        # runner.specs()
        tool_choice="auto",
        max_tokens=800,
        timeout=30,
    )
    return response.choices[0].message.model_dump(exclude_none=True)  # has "tool_calls" when it wants a tool


def run_agent_real(runner, question, max_steps=6):
    messages = [{"role": "user", "content": question}]
    for _ in range(max_steps):
        reply = real_model(messages, runner.specs())
        messages.append(reply)
        if not reply.get("tool_calls"):
            return reply.get("content")
        for call in reply["tool_calls"]:
            messages.append(runner.run(call))  # the same runner as above: validation, confirmation, audit
    return "stopped: step limit reached"

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Skip the schema check

In the Python runner's run, delete the two lines if found: and return self.finish(... "invalid" ...) that follow found = self.problems(...), then run.

terminal

$ Run

── what you'll see ──

tools sent to the model: get_order, refund_order, slow_report

run 1: a typo, then a refund the user approves

step 1: get_order -> {"order_id":17,"status":"shipped","total":40}

step 2: get_order -> {"order_id":"A17","status":"shipped","total":40}

step 3: refund_order -> {"refunded":20,"order_id":"A17"}

final: Refunded 20 on order A17.

run 2: the user declines

step 1: refund_order -> {"error":"the user declined this action"}

final: Understood, I did not refund order A18.

run 3: a model that never stops

step 1: slow_report -> {"error":"timed out after 1000 ms"}

step 2: slow_report -> {"error":"timed out after 1000 ms"}

step 3: slow_report -> {"error":"timed out after 1000 ms"}

final: stopped: step limit 3 reached

audit log: c1 get_order ok; c2 get_order ok; c3 refund_order ok; c4 refund_order declined; c5 slow_report timeout; c6 slow_report timeout; c7 slow_report timeout

Myth vs fact

Myth

A framework's tool helper makes my own runner unnecessary.

Fact

Frameworks give you a loop. Validation, confirmation, limits and logs for your tools are still your decisions, and they are easier to apply when you understand the loop.

Myth

Once the scripted version works, the real model will behave the same.

Fact

A real model makes different mistakes: other argument shapes, extra calls, odd retries. Run the same scenarios against it, then add them to your tests (Phase 9).

Myth

Tool safety belongs in the prompt.

Fact

The prompt can ask. Only code can enforce: the registry, the schema, the confirmation, the limits.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Make the runner a thin, well-tested layer between the model and your business code, and unit-test it with the scripted model: bad JSON, wrong type, unknown tool, decline, exception, timeout, step limit. These are the cases real models produce.
▸
Add a total budget to the loop (steps, seconds, tokens, dollars) and make it configurable per route. A support chat and a batch job want different limits.
▸
Emit one trace per request that includes each step's tool, outcome and duration (Phase 11). Alert on rising invalid, refused or timeout counts: they show a prompt, schema or tool that drifted.

Remember this

1
Recipe: the registry and schema. When to use it: always, for every tool. What it guards against: the model calling something that does not exist, or you forgetting what each tool needs. How to adapt it: register(name, description, properties, required, fn, write, timeout) keeps everything about a tool in one place and produces the tools list for the request (specs()), so the schema the model sees and the schema you validate against are the same object.
2
Recipe: validate before you run. When to use it: every call. What it guards against: wrong types, missing or extra fields, bad JSON (Topic 7.4). How to adapt it: swap the small hand-written problems for Pydantic, Zod or a JSON Schema library. Keep returning a list of messages: they go straight back to the model.
3
Recipe: errors are tool messages. When to use it: every failure. What it guards against: crashes, silent failures and leaked internals (Topic 7.5). How to adapt it: the runner turns every outcome (refused, invalid, declined, failed, timeout, ok) into one compact JSON tool message with the call's id. Failures the model can act on carry their text. Unexpected exceptions send only a generic line, and you log the real one.
4
Recipe: confirm writes, log everything. When to use it: any tool with side effects (Topic 7.7). What it guards against: wrong or hijacked actions, and the question 'what happened?' afterwards. How to adapt it: the confirm callback is a parameter, so tests pass a function, and an app passes a dialog. The audit list gets one line per call with its outcome. Send it to your real log, with arguments scrubbed of secrets.
5
Recipe: a loop with a step limit. When to use it: wherever a model can call tools more than once (Topic 7.2). What it guards against: loops that never end and cost money. How to adapt it: the loop here is 8 lines. Add a total time budget, a cap on tool calls per reply (Topic 7.6) and a token budget (Topic 6.2) as your product needs. The example shows the three endings: success, a user decline, and the limit.
6
Recipe: fake first, real second. When to use it: always build and test with the scripted model, then change one function. The polyShow blocks below show where a real model plugs in, using the OpenAI SDK against Groq in Python and JavaScript, and plain HTTP in Java. Nothing else changes: the runner never knew which model it served. Before you ship: run the same scenarios against a real model, and add them to your evaluation set (Phase 9).

Explain it without notes

What order of checks should a tool runner apply, and why?

Why is the confirmation function a parameter of the runner and not part of each tool?

How do you move from the scripted model to a real one safely?

Practice

Write summarise(audit) that takes audit lines like "c1 get_order ok" and returns a dict counting outcomes (the last word of each line).

Write specs_for(registry) where registry maps a name to a dict with description and schema, returning the tools list in the API shape, in registry order.

Trade-offs

↔
One runner for all tools gives uniform safety and becomes a central piece you must test well.
↔
Strict checks catch more mistakes and reject some calls a lenient system would have run correctly.
↔
A hand-written runner is small and transparent; a framework saves code and hides decisions you should make on purpose.

Done when you can

Before you ship: every tool is in a registry with a description, schema, risk label and deadline.
Before you ship: arguments are parsed, validated and authorised before any function runs.
Before you ship: writes need confirmation, carry idempotency keys and have limits per user.
Before you ship: every failure becomes a clear tool message, and internals stay in your logs.
Before you ship: the loop has a step limit, a total budget and a planned ending when they are hit.
Before you ship: every call is logged with its outcome, and the scripted scenarios also pass against a real model.

Command Palette

Words you'll meet

Step by step

01Step 1: register the tools

02Step 2: one entry point for every call

03Step 3: wrap it in a loop

04Step 4: three scenarios

05Step 5: read the audit log

06Step 6: plug in the real model

Try it yourself

Predict the audit log

Add a tool

Tighten a deadline

Code & diagrams

Break it on purpose

Skip the schema check

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can