PHASE 11Advanced ~22 min· topic 8 of 8

Topic 11.8

Playbook: An Instrumented Model Client

In one line

One wrapper around any model that leaves a redacted JSON log line, a trace span and a metrics record for every call, plus a small summary that checks the metrics against limits. Copy it and plug in your real client.

Think of it like this

A taxi with a meter, a dashcam and a trip log, all built in. The driver just drives. Every trip is timed, priced and recorded without anyone remembering to do it. The point of an instrumented client is that no feature developer has to remember anything: the recording is part of the vehicle.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Wrapper: A class that holds another object and adds behaviour around its calls.
Interface: The set of methods an object promises to offer, so a fake and a real one can be swapped.
Instrumented: Code that records logs, traces and metrics about what it does.
Usage: The token counts that a provider returns with each reply.
Request id: An id passed from the entry point through every step so records can be joined.
Limit: A threshold that a metric must stay under, kept in one place.

Step by step

01Step 1: define the interface

The fake and the real model both offer complete(prompt) and return token counts and latency. The wrapper depends only on this, so it can wrap either.

model.pywhole filepython

class FakeModel:
    def complete(self, prompt):
        return {"tokens_in": 1200, "tokens_out": 300, "latency_ms": 700}

02Step 2: wrap the call

ObservedModel.complete remembers the start time, the next span id and the model name, calls the model inside a try, and converts a provider error into an error record instead of losing it.

03Step 3: write the three signals in one place

After the call (success or failure) it advances the simulated clock, appends a metrics record, appends a span line and appends a log line with a redacted prompt. This is the whole recipe: three appends in one function.

Step 3: write the three signals in one placediagram

Rendering diagram…

04Step 4: re-raise so the caller can decide

The wrapper records the failure and then raises ProviderError again. Retries, fallbacks and error messages stay in the code that knows what to do. Recording and policy are separate jobs.

05Step 5: summarise and check limits

The records from the wrapper feed the summary function: error rate, p95 latency and cost per request. The limits dictionary is the only place to change a threshold, and the output says which checks need attention.

06Step 6: plug in the real model

Replace the fake with the real client from the code block below. Run the same four requests. The log lines, spans and records have the same shape, so dashboards and alerts built on the fake keep working.

Try it yourself

1
Add a field to the log line
Add finish_reason to the fake's replies (stop or length) and to the log line. Make one of the four calls length. What metric would you add to catch that?
2
Redact something new
Add a prompt that contains a 10-digit phone number. Add a phone pattern to the wrapper's redactor and check that the log line hides it. Where in the code should the pattern list live so that every feature shares it?
3
Tighten a limit
In the limits example, set p95_latency_ms to 3,100. Does it alert? Set it to 3,099. Why does the rule use > and not >=, and which would you choose for a real SLO?

Code & diagrams

Recipe: the observed model wrapper New tab

Four scripted calls, one of them a 503. Each call leaves a JSON log line (prompt redacted), a span line and a record. Prices are example numbers.

Expected output

{"ts":0,"event":"llm_call","request_id":"r1","model":"small","status":"ok","tokens_in":1200,"tokens_out":300,"latency_ms":700,"prompt":"Summarise my invoice"}
{"ts":700,"event":"llm_call","request_id":"r2","model":"small","status":"error","tokens_in":8,"tokens_out":0,"latency_ms":2000,"prompt":"Email [EMAIL] the report"}
{"ts":2700,"event":"llm_call","request_id":"r3","model":"small","status":"ok","tokens_in":40,"tokens_out":10,"latency_ms":250,"prompt":"Translate hello"}
{"ts":2950,"event":"llm_call","request_id":"r4","model":"small","status":"ok","tokens_in":1000,"tokens_out":200,"latency_ms":650,"prompt":"Summarise my receipt"}
span-1 model_call r1 700 ms ok
span-2 model_call r2 2000 ms error
span-3 model_call r3 250 ms ok
span-4 model_call r4 650 ms ok
calls=4 errors=1 tokens_in=2248 tokens_out=510 cost=0.001889 dollars

Recipe: metrics against limits New tab

Ten records summarised and compared with limits. Cost is in micro-dollars (1,000,000 = 1 dollar) using example prices.

Expected output

error_rate_percent: 10 (limit 15) -> ok
p95_latency_ms: 3100 (limit 2500) -> ALERT
cost_per_request_micro_dollars: 634 (limit 900) -> ok
1 of 3 checks need attention

Where the real model plugs inpython

Not run here: needs your own GROQ_API_KEY. Same interface as the fake. The Java version uses Jackson and java.net.http.HttpClient; declare throws Exception on your interface because the real call has checked exceptions.

import os
import time

import openai
from openai import OpenAI


class ProviderError(Exception):
    def __init__(self, message, latency_ms):
        super().__init__(message)
        self.latency_ms = latency_ms


class GroqModel:
    """Same interface as the FakeModel in the example: complete(prompt) -> usage and latency."""

    def __init__(self):
        self.client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.environ["GROQ_API_KEY"])

    def complete(self, prompt):
        started = time.perf_counter()
        try:
            reply = self.client.chat.completions.create(
                model="openai/gpt-oss-20b", max_tokens=300, messages=[{"role": "user", "content": prompt}]
            )
        except openai.APIError as e:
            raise ProviderError(str(e), int((time.perf_counter() - started) * 1000))
        return {
            "tokens_in": reply.usage.prompt_tokens,
            "tokens_out": reply.usage.completion_tokens,
            "latency_ms": int((time.perf_counter() - started) * 1000),
        }

# observed = ObservedModel(GroqModel(), "openai/gpt-oss-20b")  # nothing else in the wrapper changes

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Raise before recording

In the wrapper's except ProviderError block (Python), add raise as the first line, before error = str(e).

terminal

$ Run

── what you'll see ──

{"ts":0,"event":"llm_call","request_id":"r1","model":"small","status":"ok","tokens_in":1200,"tokens_out":300,"latency_ms":700,"prompt":"Summarise my invoice"}

{"ts":700,"event":"llm_call","request_id":"r3","model":"small","status":"ok","tokens_in":40,"tokens_out":10,"latency_ms":250,"prompt":"Translate hello"}

{"ts":950,"event":"llm_call","request_id":"r4","model":"small","status":"ok","tokens_in":1000,"tokens_out":200,"latency_ms":650,"prompt":"Summarise my receipt"}

span-1 model_call r1 700 ms ok

span-3 model_call r3 250 ms ok

span-4 model_call r4 650 ms ok

calls=3 errors=0 tokens_in=2240 tokens_out=510 cost=0.001885 dollars

Myth vs fact

Myth

Each feature should log its own way.

Fact

Then half the calls are missing fields. One wrapper at the one place that calls the model keeps the records uniform.

Myth

The wrapper should retry too.

Fact

Retry and fallback are policy and belong in their own layer (Topics 6.8, 6.9). Compose them: the observed model can wrap the retrying model or sit inside it.

Myth

Observability slows the call down.

Fact

Appending a small record costs microseconds beside a call of hundreds of milliseconds. Ship records off the request path (a queue or batch exporter), not inside it.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Order the layers on purpose: observe outside the retry layer to see what the caller experienced, or inside it to see each attempt. Many teams do both with different names.
▸
Make the log schema a versioned contract (schema: 1): dashboards and alerts depend on field names, and changing a name silently breaks them.
▸
Put a test around the wrapper that feeds a fake provider error, a very long prompt and a prompt full of fake secrets, and asserts that one record is written and no secret survives.

Remember this

1
Recipe: the observed model wrapper. When to use it: always, around the single place where your code calls a model. What it guards against: calls that leave no trace, and each feature inventing its own logging. How to adapt it: give the wrapper the same interface as your model (complete(prompt)). Keep the fake and the real client interchangeable. Write the log line, the span and the metrics record in one place (the first example).
2
Recipe: record failures too. When to use it: always. What it guards against: dashboards that only show successes. The wrapper writes the record in the error path before it re-raises, so the failing call is counted, logged and timed. The caller still decides what to do (retry, fall back); the wrapper does not hide the error.
3
Recipe: redact before the write. When to use it: whenever a prompt or reply goes into a log. What it guards against: personal data and secrets in logs (Topic 11.2). How to adapt it: keep the pattern list in one function, test it with fake data, and prefer logging sizes and ids for sensitive features.
4
Recipe: estimate when the provider tells you nothing. When to use it: when a call fails and no usage comes back. What it guards against: failed calls that look free on the dashboard. The wrapper estimates input tokens from the prompt length (about 4 characters per token) and records zero output tokens, marked as an estimate in your own docs. Replace it with the provider's usage whenever one is returned.
5
Recipe: metrics against limits. When to use it: for release checks and alerts. What it guards against: limits scattered through the code. The second example keeps the error rate, p95 latency and cost per request limits in one dictionary and checks the summary of the records against them, with the nearest-rank percentile from Topic 11.4.
6
Recipe: ids that join everything. When to use it: always. A request id on the log line, the span name and the record lets you move from an alert to a trace to the log to the user's feedback (Topic 11.6). Pass the id in from the entry point; never generate a new one inside the wrapper.
7
Where the real SDK plugs in. The block below shows a real client behind the same interface for Groq's OpenAI-compatible API. It reads token counts from the provider's usage, times the call, and raises the same ProviderError, so nothing else in the wrapper changes. For production, replace the hand-made span and metrics with OpenTelemetry and Prometheus (Topics 11.3 and 11.4).

Explain it without notes

Why wrap the model in one place instead of logging at each call site?

Why should the wrapper record a failed call and then re-raise the error?

How do you make sure instrumentation does not leak private data?

Practice

Write a Recorder class with a method record(status, latency_ms) and a method summary() that returns "calls=N errors=N". Use it with three calls: ok, error, ok.

Write check_limits(summary, limits) where both are dicts; return a sorted list of the names whose summary value is greater than the limit.

Trade-offs

↔
A thin wrapper is easy to read and test; a framework is richer and adds a dependency.
↔
Synchronous recording is simple and adds a little latency; asynchronous export is faster and can lose records on a crash.
↔
Recording more fields helps investigations and enlarges storage and privacy exposure.

Done when you can

Before you ship: every model call goes through one instrumented wrapper.
Before you ship: failed calls are logged, traced and counted, then re-raised.
Before you ship: prompts and replies are redacted before logging, and full text is sampled with a retention limit.
Before you ship: every record carries a request id, model, prompt version, tokens, latency and status.
Before you ship: dashboards show p50, p95, TTFT, errors by cause, tokens, cost and cache hit rate, with deploy markers.
Before you ship: alerts have a minimum volume, a window rule, an owner and a runbook, plus a hard cost cap.

Command Palette

Words you'll meet

Step by step

01Step 1: define the interface

02Step 2: wrap the call

03Step 3: write the three signals in one place

04Step 4: re-raise so the caller can decide

05Step 5: summarise and check limits

06Step 6: plug in the real model

Try it yourself

Add a field to the log line

Redact something new

Tighten a limit

Code & diagrams

Break it on purpose

Raise before recording

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can