PHASE 4Intermediate ~21 min· topic 5 of 8

Topic 4.5

Validating Output

In one line

Parsing tells you the text is JSON. Validation tells you it has the right shape and allowed values. Write a small validator by hand first so you know what libraries do, then use Pydantic, Jackson with Bean Validation or Zod, and write errors a model can act on.

Think of it like this

The border check at an airport. Your passport is read (parsing), then an officer checks it is real, not expired, and that you have the right visa (validation). Reading the passport tells nothing about whether you may enter. Validation is the check that stands between model text and the rest of your program.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Validation: Checking that data has the right shape and allowed values before using it.
Untrusted input: Data from outside your code, such as a user or a model. It may be wrong in any way.
Pydantic: A Python library that validates data against a class definition.
Bean Validation: A Java standard for rules written as annotations such as @NotNull and @Min.
Jackson: The most used Java library for reading and writing JSON.
Zod: A JavaScript and TypeScript library that validates data against a schema.
Coercion: Converting a value to the wanted type automatically, such as the string "4" to the number 4.

Step by step

01Step 1: parse, then check

The example has a schema (urgency is low, medium or high; score is an integer from 1 to 5; tags is a list of strings; all three required) and four replies. First each reply is parsed. Only then is it validated. Parsing and validating are different steps with different errors.

02Step 2: one function walks the schema

validate(value, schema, path) looks at the type, then the enum, then (for integers) the range, then (for objects) the required keys and each property, then (for arrays) each item. Objects and arrays call validate again for their children with a longer path such as $.tags[1]. It returns a list, so one run reports every problem.

main.pywhole filepython

if got == "array":
    for i, item in enumerate(value):
        errors += validate(item, schema["items"], f"{path}[{i}]")

Step 2: one function walks the schemadiagram

Rendering diagram…

03Step 3: read the output

Reply 1 is valid. Reply 2 has three problems at three places: the enum, the type of score, and the type of an item in tags. Reply 3 is missing tags and has a score of 9. Reply 4 is not JSON at all. Each message says where, what was found and what was wanted.

04Step 4: bring in a library

The block after the example shows the same contract in Pydantic, Jackson with Bean Validation and Zod. They give you typed objects (Ticket) instead of dictionaries, which your editor can check, and they can generate the JSON Schema for Topic 4.4. The ideas are the same ones the hand-written code uses.

05Step 5: turn problems into feedback

Each library lists problems differently (loc and msg in Pydantic, getPropertyPath() and getMessage() in Bean Validation, path and message in Zod). Join them into plain sentences, one per problem, as the hand-written version already does. Topic 4.6 sends those sentences back to the model.

Try it yourself

1
Predict the problems
For reply 2, write down the three problems you expect, with their paths, before you run. Then compare with the output, including the order.
2
Add a rule
Add "minLength" support: for a string with minLength, report $.name: is shorter than the minimum length 3. Add a name property to the schema and a reply that breaks it.
3
Make it strict
Add support for additionalProperties: false: report $.plann: unexpected field for any key that is not in properties. Try a reply with a typo.

Code & diagrams

A hand-written validator that reports every problem New tab

Checks type, enum, range, required keys, properties and array items. Java uses the course's Json helper; real Java projects use Jackson (and Bean Validation, shown next).

Expected output

reply 1: valid
reply 2: 3 problem(s)
  - $.urgency: "urgent" is not one of low, medium, high
  - $.score: expected integer, got string
  - $.tags[1]: expected string, got integer
reply 3: 2 problem(s)
  - $.tags: required field is missing
  - $.score: 9 is greater than the maximum 5
reply 4: not valid JSON

The same contract with Pydantic, Jackson + Bean Validation and Zodpython

Not run here: these libraries are not available in the browser. Check each library's current version and API in its documentation before you copy.

# pip install pydantic
from typing import Literal

from pydantic import BaseModel, ConfigDict, Field, ValidationError


class Ticket(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)  # no unknown keys, and "4" is not accepted as 4

    urgency: Literal["low", "medium", "high"]
    score: int = Field(ge=1, le=5)
    tags: list[str]


def read_ticket(reply_text: str):
    try:
        return Ticket.model_validate_json(reply_text), []
    except ValidationError as e:
        # each error has a location (loc), a message (msg) and a type; turn them into sentences for the model
        problems = [".".join(str(p) for p in err["loc"]) + ": " + err["msg"] for err in e.errors()]
        return None, problems


ticket, problems = read_ticket('{"urgency": "urgent", "score": 9, "tags": ["login"]}')
print(problems)
print(Ticket.model_json_schema())  # the JSON Schema to send to the provider (Topic 4.4)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget the required list

In the Python example delete "required":["urgency","score","tags"], from the schema text and run it.

terminal

$ Run

── what you'll see ──

reply 1: valid

reply 2: 3 problem(s)

- $.urgency: "urgent" is not one of low, medium, high

- $.score: expected integer, got string

- $.tags[1]: expected string, got integer

reply 3: 1 problem(s)

- $.score: 9 is greater than the maximum 5

reply 4: not valid JSON

Myth vs fact

Myth

If the JSON parses, the data is fine.

Fact

Parsing checks syntax. Types, ranges, required keys and allowed values are all still unchecked.

Myth

Validation libraries are only for big projects.

Fact

A 40-line function does the job for a small one. Use a library when the contract grows or you want typed objects.

Myth

Auto-conversion is always helpful.

Fact

Turning "4" into 4 hides model drift. For model output prefer strict checks and count how often reality differs.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Report all problems at once. A model can fix three problems in one retry, but needs three retries if you reveal one per attempt.
▸
Keep the error text short and free of the user's data when it goes into logs; log the path and the rule, not the value, if the value could be private.
▸
Version the model class or schema with the prompt, and keep a small set of good and bad sample replies as unit tests for it (Topic 3.8, Phase 9).

Remember this

1
Why this exists. Even with JSON mode or a schema (Topic 4.3), data from a model is untrusted input, like data from a web form. It can be missing a key, have a string where you need a number, a score of 9 on a scale of 1 to 5, or a label you never listed. If it flows on unchecked, the failure appears far from its cause: a crash in a database insert, a wrong total in a report, a ticket in the wrong queue.
2
Validate at the boundary, once. Right after parsing, check the data against the contract and produce one of two things: a value your program can trust, or a list of problems. After that point the rest of the code can assume the data is right. The hand-written validator in the example does this in about 40 lines. It checks the type, the enum, the range of integers, the required keys, the properties and the items of arrays. It reports every problem, not only the first.
3
Write errors for a reader that can act on them. $.score: 9 is greater than the maximum 5 tells a model (or an engineer) exactly what to fix: the place, the value and the rule. validation failed tells nothing. Good messages are also what makes the repair loop in Topic 4.6 work, because they go back to the model as feedback. Say where, what was found and what was wanted.
4
Libraries do the same, with less code. In Python, Pydantic defines a class and validates JSON against it (Ticket.model_validate_json(text)). It raises a ValidationError whose errors() list gives a location, a message and a type for each problem. In Java, Jackson reads JSON into a record or class, and Bean Validation (@NotNull, @Min, @Max, checked by a Validator, with Hibernate Validator as the usual engine) checks the rules. In JavaScript, Zod defines a schema, and safeParse returns either the data or a list of issues. Recall from Topic 4.4: a schema lists the keys, types and allowed values. The shapes below are shown, not run, because they need packages that cannot load in the browser.
5
Strict or lenient, decide on purpose. Pydantic by default converts where it can (the string "4" becomes the integer 4). That is friendly for form data and risky for model output, because you stop noticing when the model drifts. Zod's z.number() rejects strings, and Jackson turns a missing int into 0 unless you use Integer with @NotNull. Whatever the library, test it with a known bad reply and read what happens.
6
Validation is not truth. A valid object can still be wrong: the label is allowed but incorrect, the date has the right format but isn't the date in the text. Business checks (does this customer id exist?) come after shape checks, and for facts in the source text there is the evidence check in Topic 4.7. Also remember what a validator costs: microseconds. Skipping it to save time is never worth it.

Explain it without notes

Why must you validate model output even when you use JSON mode or a strict schema?

What makes a validation error message useful to a model?

In what way can library defaults hide problems with model output?

Where in a program should validation happen?

Practice

Write check_score(data) that returns a list of problems for a dictionary: score must exist, be an integer and be between 1 and 5. Use messages like score: missing.

Write describe(problems) that turns a list of problems into the feedback text Please fix: p1; p2 or an empty string when there are none.

Trade-offs

↔
A hand-written validator has no dependencies and is easy to read, but you maintain every rule; libraries give typed objects and generated schemas at the cost of a dependency.
↔
Strict validation catches drift early but rejects more replies, which means more retries.
↔
Reporting every problem costs a few more lines than failing on the first, and saves a retry.

Done when you can

Done when you can explain the difference between parsing and validating.
Done when you can write a validator that reports type, enum, range, missing key and array item problems.
Done when you can write an error message a model can act on.
Done when you can show the same contract in Pydantic, Bean Validation and Zod.
Done when you can name a library default that hides model drift.

Command Palette

Words you'll meet

Step by step

01Step 1: parse, then check

02Step 2: one function walks the schema

03Step 3: read the output

04Step 4: bring in a library

05Step 5: turn problems into feedback

Try it yourself

Predict the problems

Add a rule

Make it strict

Code & diagrams

Break it on purpose

Forget the required list

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can