PHASE 4Intermediate ~23 min· topic 8 of 8

Topic 4.8

Playbook: Typed LLM Calls

In one line

The recipes of the whole phase in one place: parse a reply into a value or a list of problems, wrap it in one typed-call function that repairs and falls back, and count what happens. Copy these into any LLM feature.

Think of it like this

The quality line at a factory. Every part is inspected (validation). A part with a fixable flaw goes back for rework with a note saying what is wrong (repair). A part that fails twice goes in the reject bin (fallback). And a clipboard counts how many parts took each path, because a rising reject count means the machine needs attention. The playbook is that line, as code.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Typed call: A model call whose result is a checked value of a known shape, not raw text.
Recipe: A small piece of code and a rule of use that you can copy into a project.
Rework: Sending a flawed result back with a note about what is wrong (the repair step).
Reject: Giving up on a request and returning a flagged safe value (the fallback step).
Repair rate: The share of requests that needed more than one attempt.
Single source of truth: One definition (here the schema or type) that every other part is generated from.

Step by step

01Step 1: parse_reply turns text into a value or problems

The first recipe combines the extractor of Topic 4.2 and the validator of Topic 4.5 around a spec. Five cases show the paths: clean, chatter around the JSON, a missing field, a wrong type and a bad label, and a cut-off reply. It never throws on bad model text.

main.pywhole filepython

value, problems = parse_reply(text, SPEC)
if value is None:
    # problems is a list of sentences: where, what was found, what was wanted
    ...

02Step 2: typed_call wraps the loop

The second recipe is the repair loop of Topic 4.6 as one function: call, parse, return the value if it is good, otherwise append the bad reply and the rejection message, and try again until max_attempts. If every attempt fails it returns the fallback and says so.

Step 2: typed_call wraps the loopdiagram

Rendering diagram…

03Step 3: count what happens

The same function updates a small stats dictionary. In the example, five requests produce three first-try successes, one repair and one fallback, and two extra calls beyond the five minimum. Those two numbers are what you chart in production.

04Step 4: plug in the real model

The last block shows the real client in all three languages with the same shape as the fake: messages in, reply text out. JSON mode is switched on, and a reply cut off at max_tokens becomes an empty string, which fails parsing and triggers a repair. Nothing else in the recipe changes.

05Step 5: before you ship

The checklist at the bottom is the list to walk through before an LLM feature that feeds code goes live. Each line is a lesson from this phase.

Try it yourself

1
Add a field to the spec
Add ("urgent", "boolean", None) to SPEC and a case that leaves it out. Predict the problem line, then run.
2
Change max_attempts
In the typed-call example, change the 2 to 1. Predict the three counters, then run. Then set it to 3 and see whether anything improves.
3
Break the fake on purpose
Give request 4 a valid third reply. With max_attempts at 2 it still falls back. What does max_attempts 3 do, and what does it cost in extra calls?

Code & diagrams

Recipe: parse_reply, text in, value or problems out New tab

Extractor, type check, enum check in one function. Java uses the course's Json helper; real projects use Jackson.

Expected output

case 1: ok label=bug confidence=88
case 2: ok label=feature confidence=71
case 3: confidence: required field is missing
case 4: label: "refund" is not one of billing, bug, feature, unknown | confidence: expected integer, got string
case 5: no complete JSON object in the reply

Recipe: typed_call with repair, fallback and counters New tab

The model is a parameter: scripted here, a real client in production. max_attempts is 2 so the fallback case is visible.

Expected output

request 1: label=bug
request 2: label=billing
request 3: label=feature
request 4: label=unknown (fallback)
request 5: label=bug
first_try=3 repaired=1 fallback=1 calls=7
extra calls caused by repairs and fallbacks: 2

Where the real model plugs inpython

Not run here: it needs a network call and your own GROQ_API_KEY. Same interface as the fake, so typed_call doesn't change.

# pip install openai      (Groq, and many other providers, speak the OpenAI-compatible API)
import os

from openai import OpenAI

client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.environ["GROQ_API_KEY"], timeout=30)


def model(messages):
    """The real model behind the same interface as scripted(): messages in, reply text out."""
    r = client.chat.completions.create(
        model="openai/gpt-oss-20b",
        messages=[{"role": role, "content": text} for role, text in messages],
        response_format={"type": "json_object"},  # JSON mode: say "JSON" in the prompt too
        max_tokens=500,
    )
    choice = r.choices[0]
    if choice.finish_reason == "length":
        return ""  # cut off: an empty reply fails parse() and triggers a repair attempt
    return choice.message.content or ""


# typed_call(model, "Reply in JSON with the key label. Classify this ticket: ...", parse, 2, stats)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Allow only one attempt in typed_call

In the Python typed-call example change the 2 passed as max_attempts in the loop at the bottom to 1.

terminal

$ Run

── what you'll see ──

request 1: label=bug

request 2: label=unknown (fallback)

request 3: label=feature

request 4: label=unknown (fallback)

request 5: label=bug

first_try=3 repaired=0 fallback=2 calls=5

extra calls caused by repairs and fallbacks: 0

Myth vs fact

Myth

A typed call is just json.loads plus a try/except.

Fact

That misses chatter, wrong fields, wrong values, truncation, repair and counting. The recipe is the difference between a demo and a feature.

Myth

If the fallback rate is low, nothing is wrong.

Fact

A low fallback rate with a high repair rate means the prompt is fragile and costs more than it should. Watch both.

Myth

The typed-call function is the same for every task.

Fact

The loop is, but the spec, the fallback value and the attempts depend on the task and on how harmful a wrong answer is.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Treat parse as a plug-in. The same typed_call serves classification, extraction and planning; only the spec and fallback change. Test typed_call once with a fake and each parse with sample replies.
▸
Log the raw reply and the problems for every repaired or fallback request, with a request id and the prompt version. When the rate rises you can look at real examples instead of guessing.
▸
Budget the worst case: attempts times the largest request cost plus a deadline for the whole call. Retrying content failures and retrying network failures are separate counters and separate caps (Topics 6.8, 6.9).

Remember this

1
Recipe: parse_reply. When to use it: every time model text becomes data. What it guards against: chatter around the JSON, malformed JSON, cut-off replies, missing fields, wrong types and values outside a list. How to adapt it: change the spec (a list of name, type and allowed values), or replace it with Pydantic, Jackson with Bean Validation or Zod (Topic 4.5). Keep the contract: it returns a value or a list of problems, never both and never an exception for bad model text.
2
Recipe: typed_call. When to use it: any call where the answer feeds code. What it guards against: one-off bad replies (by repairing with the problem list), endless loops (a maximum on attempts), and a crash when the model keeps failing (a flagged fallback value). How to adapt it: pass your own parse, your own max_attempts (2 or 3), and a fallback that is safe for your product. Make the model call a parameter so tests can use a fake and production a real client.
3
Recipe: the three counters. When to use it: always, from the first deployment. What it guards against: silent drift. first_try, repaired and fallback tell you at a glance how healthy the prompt is. How to adapt it: add the model name and prompt version as labels, send the counters to your metrics system (Phase 11), and alert when the repair or fallback rate rises.
4
Recipe: ask for null, check the source. When to use it: extraction. What it guards against: invented values. How to adapt it: nullable fields, the sentence "if it is not in the text, use null", and the evidence check of Topic 4.7 on every value that should be copied from the source.
5
Recipe: one schema, many uses. When to use it: when the shape is used in more than one place. What it guards against: a prompt, a parser and documentation that disagree. How to adapt it: define the type once (Pydantic model, Java record, Zod schema) and derive the JSON Schema from it. Send it to the provider if the model supports strict output (Topic 4.3). Validate with the same definition.
6
Recipe: unknown is an answer. When to use it: classification and routing. What it guards against: forced wrong labels. How to adapt it: add unknown with a rule, a confidence threshold chosen from data, and a queue where unknowns are handled by a person.

Explain it without notes

Describe the typed-call recipe step by step.

Which counters would you put on a typed call, and what does each tell you?

How would you adapt this phase's recipes to a model that does not offer JSON mode?

Where does validation end and evaluation (Phase 9) begin?

Practice

Write summarise_stats(stats, total) that returns first_try=A repaired=B fallback=C followed by repair rate: B of total as two lines, given a stats dictionary with those keys.

Write safe_label(parse_result, allowed) that takes (value, problems) and returns value['label'] if the value exists and its label is in allowed, otherwise "unknown".

Trade-offs

↔
A strict spec rejects more replies and gives cleaner data; a loose spec costs fewer repairs and lets more errors through.
↔
More attempts mean higher success and higher worst-case cost and latency.
↔
Counting every outcome costs a little code and is the only way to see quiet failures.

Done when you can

Before you ship: the shape is defined once (a type or schema) and the prompt describes the fields in words.
Before you ship: every reply goes through extract, parse and validate, and bad model text never raises.
Before you ship: there is a maximum number of attempts, feedback that names the problem, and a flagged fallback.
Before you ship: fields that may be absent are nullable, classifiers have unknown, and extracted facts are checked against the source.
Before you ship: finish_reason, first_try, repaired and fallback are counted and charted by prompt version.
Before you ship: a small set of good and bad sample replies is in tests, and a labelled set of real inputs is ready for evaluation.

Command Palette

Words you'll meet

Step by step

01Step 1: parse_reply turns text into a value or problems

02Step 2: typed_call wraps the loop

03Step 3: count what happens

04Step 4: plug in the real model

05Step 5: before you ship

Try it yourself

Add a field to the spec

Change max_attempts

Break the fake on purpose

Code & diagrams

Break it on purpose

Allow only one attempt in typed_call

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can