PHASE 0Beginner ~21 min· topic 7 of 7

Topic 0.7

Playbook: Your First LLM Feature

In one line

Two recipes to start every LLM feature: a fake model behind the same interface as the real one, so the feature is testable from day one, and a meter that counts tokens, estimates cost and refuses calls over budget. Plus the checklist to run before you ship.

Think of it like this

A pilot's pre-flight checklist. Experienced pilots still go through it every time, because the cost of forgetting one item is too high and a list is cheaper than memory. This playbook is the checklist and the two tools you set up before writing the first prompt.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Interface: The agreed shape of a component (its method names, inputs and outputs), so different implementations can be swapped.
Playbook: A set of tested recipes and checklists you reuse instead of reinventing them each time.
Usage: The part of an API reply that reports how many input and output tokens the call used.
Budget guard: Code that refuses a call when it could push spending past a limit.

Step by step

01Step 1: write the interface first

Decide the one method your feature calls. Keep it small: messages in, a reply with text and finish_reason out. Everything else (provider, model name, retries) hides behind it.

model.pywhole filepython

class ChatModel:
    def complete(self, messages, max_tokens=256):
        """Returns {"text": str, "finish_reason": str, "usage": {"input": int, "output": int}}."""
        raise NotImplementedError

02Step 2: test the feature against the fake

The first recipe tests summarise with three scripted replies: normal, cut off, and empty. The feature handles each one. The recorded requests let you check the prompt you actually sent, which is how you catch a missing system message in a test instead of in production.

03Step 3: wrap the model in a meter

The second recipe wraps the model. Each call adds its reported usage. Before each call it asks: with this input, and if the reply used every output token I allow, could we pass the budget? Each call really costs 0.00016 dollars, but the worst case adds 0.00022. Call 6 could pass 0.001, so it is refused before any money is spent, and total spending (0.00080) stays under the budget.

Step 3: wrap the model in a meterdiagram

Rendering diagram…

04Step 4: plug in the real client

The last block shows the real client for Groq's OpenAI-compatible API in each language. It has the same complete method, so summarise(OpenAICompatibleModel(...), article) and MeteredModel(OpenAICompatibleModel(...), budget) work without changing the feature. Phase 2 rebuilds this client properly: timeouts, error classes, retries.

05Step 5: run the checklist

Before shipping, go through the checklist below. Each later phase adds items to it and a recipe to this playbook.

Try it yourself

1
Add a failing case
Add a fourth scripted reply ("Rain all week.", "content_filter") and a fourth case name. What does summarise return? Add a check so any finish_reason other than stop is reported.
2
Find the last affordable call
In the meter recipe, change the budget to 0.0005. Predict which call is refused (each call costs 0.00016, and the worst-case check adds 0.00022 on top of what is already spent), then run.
3
Copy a lab as code
Open a lab in Topic 0.6, choose Copy as code and your language, and compare it with the real client above. Which fields match?

Code & diagrams

Recipe: a fake-first model client New tab

The feature is tested against cut-off and empty replies before any real model is involved.

Expected output

normal reply   -> Rain is expected all week in Pune.
cut-off reply  -> ERROR: summary was cut off (raise max_tokens or ask for less)
empty reply    -> ERROR: empty summary
requests recorded: 3, system prompt: Summarise the user's text in one sentence.

Recipe: a usage and cost meter with a budget New tab

Prices are example numbers. Read your provider's price page and keep prices in configuration.

Expected output

call 1: ok, total so far 0.000160 dollars
call 2: ok, total so far 0.000320 dollars
call 3: ok, total so far 0.000480 dollars
call 4: ok, total so far 0.000640 dollars
call 5: ok, total so far 0.000800 dollars
call 6: refused (budget 0.001000 would be passed)
calls=5 tokens_in=5000 tokens_out=750
cost per 1,000 such calls: 0.1600 dollars

Where the real model plugs inpython

Not run here: it needs a network call and your own GROQ_API_KEY. Same interface as the fake, so the feature code doesn't change.

# pip install openai      (Groq, and many other providers, speak the OpenAI-compatible API)
import os

from openai import OpenAI


class OpenAICompatibleModel:
    """The real model behind the same complete() interface as the FakeModel recipe."""

    def __init__(self, model, base_url="https://api.groq.com/openai/v1"):
        # the key comes from the environment, never from code; timeout in seconds
        self.client = OpenAI(base_url=base_url, api_key=os.environ["GROQ_API_KEY"], timeout=30)
        self.model = model

    def complete(self, messages, max_tokens=256):
        r = self.client.chat.completions.create(model=self.model, messages=messages, max_tokens=max_tokens)
        choice = r.choices[0]
        return {
            "text": choice.message.content or "",
            "finish_reason": choice.finish_reason,
            "usage": {"input": r.usage.prompt_tokens, "output": r.usage.completion_tokens},
        }


# summarise(OpenAICompatibleModel("openai/gpt-oss-20b"), article) now calls a real model

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Check the budget after the call

In the meter recipe, move the budget check to after self.model.complete(...) and the token counting, and compare self.cost() with the budget instead of the worst case. Run it.

terminal

$ Run

── what you'll see ──

call 7: refused (budget 0.001000 would be passed)

Myth vs fact

Myth

Fakes are only for tests.

Fact

They also let you build the UI and the whole pipeline before you have a key, a budget or a chosen provider.

Myth

Cost tracking can wait until the bill arrives.

Fact

By then a loop or a long prompt has already cost the money. Meter from the first real call.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Make the meter and the real client both implement the same interface and compose them: Metered(Retrying(Cached(RealModel))). Each concern stays small and testable (Phase 6, Phase 12).
▸
Count tokens from the provider's usage, not your own estimate: tokenizers differ by model, and reasoning tokens are invisible in the text (Topic 1.2).
▸
Keep a per-feature cost dashboard from day one; product decisions (which model, how much history) are cost decisions.

Remember this

1
Recipe: a fake-first model client. When to use it: at the start of every feature. Define the interface your feature needs (complete(messages, max_tokens) -> {text, finish_reason}) and write a FakeModel that returns scripted replies and records every request. Build and test the feature against the fake: normal reply, cut-off reply, empty reply, error. Then plug in the real client behind the same interface (the last block below). What it guards against: features that only work on the happy path, and tests that need a key and money to run. How to adapt it: add usage, tool calls or streaming to the reply shape as your feature needs them.
2
Recipe: a usage and cost meter. When to use it: around every real model client, in every environment. It wraps any model with the same interface, counts calls and tokens from each reply's usage, turns them into cost with your provider's prices, and refuses a call that could pass a budget (it checks the worst case: this request's estimated input plus every allowed output token). What it guards against: surprise bills, runaway loops, and not knowing what a feature costs. How to adapt it: keep budgets per user, per day or per feature; export the counters as metrics (Topic 11.4); set prices from config, never from memory.
3
Checklist: is an LLM the right tool? Could plain code do it (Topic 0.3)? What happens when the model is wrong, and who notices? Is there a check code can do on the answer?
4
Checklist: the shape. What exactly goes in (which messages, which data) and what exactly comes out (free text, or fields your code reads)? Write three example inputs and the outputs you want before writing a prompt. Those become your first evaluation cases (Phase 9).
5
Checklist: the limits. A timeout on every call, max_tokens on every call, a budget, a plan for 429s (Phase 6), and a check of finish_reason on every reply. The key comes from the environment, lives only on the server, and never appears in logs.
6
Checklist: seeing it work. Log every call's model, tokens, latency, cost and outcome (not the user's raw text unless redacted). Count fallbacks and errors. If you can't answer "how many calls failed yesterday, and why?", the feature isn't ready (Phase 11).

Explain it without notes

Why build an LLM feature against a fake model first?

Why does the budget guard check the worst case before the call instead of the actual cost after it?

Practice

Extend FakeModel so a scripted reply can be an exception (for example a RateLimitError instance) that complete raises instead of returning, and test that summarise lets it propagate.

Add a report() method to the meter that returns "calls=N tokens_in=N tokens_out=N cost=$X" with the cost to 6 decimals.

Trade-offs

↔
An interface from day one costs a few lines; skipping it costs a rewrite when you change provider or add caching.
↔
A strict budget can refuse legitimate work; pair it with alerts and a way to raise it, not silent failure.

Done when you can

Before you ship: plain code can't do this job better (Topic 0.3).
Before you ship: three example inputs and wanted outputs are written down.
Before you ship: the feature is tested against a fake with normal, cut-off, empty and error replies.
Before you ship: every call has a timeout and max_tokens, and checks finish_reason.
Before you ship: the key is in the server's environment only, and calls are metered against a budget.
Before you ship: every call is logged with model, tokens, latency, cost and outcome.

Command Palette