PHASE 13Advanced ~26 min· topic 7 of 8

Topic 13.7

Evals a Frontend Team Owns

In one line

An eval is a saved set of cases with automatic checks. A frontend team owns the UI-critical ones: does the reply parse, fit, avoid raw HTML, mark refusals. They run on scripted replies in every build and on real replies when the prompt or model changes.

Think of it like this

A school exam with a fixed question paper and a marking key. The same paper is used for every batch, so you can say whether this year's batch did better or worse. Nobody marks by gut feeling, and a changed paper is a deliberate act. An eval is a fixed question paper for your AI feature.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Eval: A saved set of cases with automatic checks that you re-run when something changes.
Check: A function that looks at a reply and says pass or fail.
Fixture: A saved input or reply used in tests and evals.
Regression: Something that used to work and broke.
Schema validity: Whether a structured reply matches the shape the UI needs.
Refusal: A reply where the model declined. Some providers mark a blocked reply with the finish reason content-filter. A model that declines in plain words may finish with stop, so the UI needs an agreed flag to show its own message.
Pass rate: The share of cases or runs that passed. The honest way to report live evals.
CI gate: A step in the build that fails the build when a check fails.

Step by step

01Cases, checks, a verdict

The first example defines the shape the order card needs with zod, six recorded replies, and for each one the expected verdict. run() asks the stand-in model for an object with Output.object({ schema }) and reports valid, or invalid with the SDK's error name (AI_NoObjectGeneratedError). The last line is what CI would use: 6 of 6 passed → CI exit code 0.

02Two prompts, same cases

The second example keeps two recorded replies per case, one from each version of the system prompt. The checks are small functions. Look at the printout: version 1 passes everything, version 2 fails six checks, and each failure names the case and the check. A refusal case is only checked for the refusal flag, since an empty text is correct there.

Two prompts, same casesdiagram

Rendering diagram…

03Run them in CI

In a real project the eval file is a script run by your test runner or a plain Node command, and CI runs it on every pull request. For live evals, use a separate job that only runs when the prompt or schema files change, with its own small budget.

.github/workflows/evals.yml (not run here)whole fileyaml

name: evals
on:
  pull_request:
    paths: ["src/prompts/**", "src/schemas/**", "evals/**"]
jobs:
  scripted:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24 }
      - run: npm ci
      - run: npm run evals:scripted   # fixtures only: free, deterministic, must be 100%
  live:
    needs: scripted
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24 }
      - run: npm ci
      - run: npm run evals:live -- --runs 3 --min-pass 0.95   # real model: a pass RATE with a small budget
        env:
          PROVIDER_API_KEY: ${{ secrets.EVAL_PROVIDER_API_KEY }}

04When a check fails

A failure has three possible causes, and the printout tells you which to suspect. The prompt or model changed and the reply got worse (version 2 above): fix the instructions or roll back. The UI changed and no longer handles a reply it used to handle: fix the code. The check is wrong or too tight: change it, in the same pull request, with a sentence explaining why.

Never "fix" a red eval by deleting the case. If a case is no longer real, say so in the commit message.

Try it yourself

1
Add a case you saw in production
In the first example add a fixture where the reply is the correct object followed by a sentence: {"orderId":"ORD-1042","status":"shipped","summary":"On its way"} Hope that helps!. Set expect to "invalid" and run it: the SDK refuses a reply that is not exactly JSON (7 of 7 pass). Then decide whether your app should instead try to rescue the JSON before giving up, and write that as a new check. Every real bug you meet becomes a fixture.
2
Soften a check
In the second example change the length limit from 200 to 400. The long "return window" v2 reply now passes the length check but still fails "ends like a finished sentence", so v2 fails 5 checks instead of 6. Thresholds are choices: write down why 200 was chosen (the bubble's width on a phone) and not just the number.
3
Make the eval fail the build
In the second example's loop, after const failures = ..., add console.log("CI exit code:", failures.length ? 1 : 0);. v1 prints 0 and v2 prints 1. In Node you would write if (failures.length) process.exitCode = 1; instead, which is what CI reads (the preview has no process, so that line would throw a ReferenceError here). Think about which of the five checks you would make blocking and which only reported.

Code & diagrams

Schema validity: six recorded replies against the shape the card needs

Each fixture states what the app should do: accept the good reply, and notice every bad one so it can fall back. The replies are saved text, so this runs the same every time and costs nothing.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

PASS good reply               → valid
PASS unknown status           → invalid (AI_NoObjectGeneratedError)
PASS missing field            → invalid (AI_NoObjectGeneratedError)
PASS prose around the JSON    → invalid (AI_NoObjectGeneratedError)
PASS summary far too long     → invalid (AI_NoObjectGeneratedError)
PASS empty reply              → invalid (AI_NoObjectGeneratedError)
6 of 6 passed → CI exit code 0

Format and refusal checks, run against two prompt versions

v1 and v2 stand for two versions of your system prompt (the instructions you send with every request). The replies are saved text. A live run would call the real model for each case, a few times each, and report a pass rate.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

prompt v1: all checks passed
prompt v2: 6 checks failed
  FAIL order status: no markdown table
  FAIL order status: ends like a finished sentence
  FAIL return window: short enough for the bubble
  FAIL return window: ends like a finished sentence
  FAIL off-topic request: refusal is flagged, not shown as an answer
  FAIL cancelled order: no raw HTML

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Accept any status string

In the first example change z.enum(["processing", "shipped", "cancelled"]) to z.string().

terminal

$ Console

── what you'll see ──

FAIL unknown status → valid

5 of 6 passed → CI exit code 1

Myth vs fact

Myth

Evals are only for the people who train models.

Fact

Any team shipping an AI feature has behaviour to protect. A frontend team's evals protect the screen: formats, schemas, refusals, lengths.

Myth

An eval must call the real model.

Fact

Scripted replies test your handling code for free on every commit. Live runs are for measuring the model, and are a second job.

Myth

All checks should block the build.

Fact

Block on what would break the screen. Track softer things as rates, because live models vary and a single bad run is not a bug.

Myth

Passing evals means the feature is good.

Fact

They show that the checked behaviours did not regress. Quality needs its own evals and real user feedback.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Version your fixtures with the schema and the prompt. When the schema changes, an old fixture may become invalid on purpose, and the commit that changes both is the review moment.
▸
Make live evals cheap: a small set, 3 runs per case, a hard budget, and a pass-rate threshold such as "at least 95% of runs" rather than all of them. If a case flips between runs, its check is probably too tight or the instruction too vague.
▸
Log every real UI parse failure (the event, not the text) and read the bucket weekly. The cases you cannot reproduce from fixtures are the ones to ask for, with the user's permission and personal data removed.
▸
For structured output, also test the partial states of a streamed object (Topic 7.4): half an object must never draw a half card.

Remember this

1
The problem this solves: Someone edits the instructions, or the provider updates the model, and the order card stops drawing because a field is missing. No unit test fails, because no code changed. You find out from users. An eval set catches that: it re-checks a fixed list of cases whenever anything changes. This topic covers the small, deterministic evals that a frontend team can own. Evals of answer *quality* (is it correct, helpful, grounded) are a model-side skill, taught in the LLM Engineering course (/llm-engineering).
2
An eval case has an input (the question or conversation), a way to get a reply, and one or more checks. A check is a function from the reply to pass or fail. The checks a frontend team owns are about the screen: does the reply parse against the schema the card needs, does it fit the bubble, is there raw HTML or a table the renderer cannot show, does it end like a finished sentence, does it mention the order that was asked about, and, for requests the assistant should decline, did the reply arrive flagged as a refusal so the UI can show its own message. How a refusal is flagged depends on your setup. Some providers report a blocked reply with the finish reason content-filter, which is what the second example checks. A model that declines in ordinary words usually finishes with stop and no flag at all, so many teams instruct it to answer with an agreed marker (for example the single word REFUSE) and check for that. Find out what your provider does, with a recorded fixture, before you write the check.
3
Two modes. With scripted replies (recorded fixtures), the eval is deterministic and free: it tests that your *UI code and parsing* handle every kind of reply, including the ugly ones. It runs on every commit. With live replies from the real model, the eval measures what the model does today. It costs money, varies between runs, and runs when the prompt, model or schema changes, or nightly. Report a pass rate, not a single pass or fail, and run each case a few times.
4
Where cases come from. Start with the failures you have already seen: every bug report becomes a case. Add the edge cases from this course (empty reply, huge reply, emoji, a reply with prose around the JSON, a refusal). Add real questions from production after removing personal data (Topic 13.5). A hundred good cases beat ten thousand random ones.
5
Schema validity is the most valuable check. If the card needs { orderId, status, summary }, run the same zod schema your UI uses. The first live example feeds six recorded replies through Output.object and expects valid for the good one and a controlled invalid for five bad ones: an unknown status, a missing field, prose around the JSON, a summary 200 characters long, and an empty reply. "Invalid" is not a failure of the eval. It means the app noticed and can fall back (Topic 7.3).
6
Compare versions. The second example evaluates two versions of a system prompt on the same cases. Version 1 passes everything. Version 2 breaks six checks: a markdown table, a long rambling answer, raw HTML, a missing refusal flag. This comparison is the point: before you ship a prompt change you see exactly which UI-critical behaviours it breaks.
7
Gate on the UI-critical ones. In CI the eval runner prints the failures and exits with a non-zero code when a UI-critical check fails, which stops the merge. Softer checks (tone, length targets) are reported and tracked, but do not block. Decide which is which before the first failure, not during it.
8
Keep them honest. Evals can pass while users are unhappy, because the checks are narrow. They are a safety net for the screen, not a measure of quality. Pair them with thumbs-down rates (Topic 13.5) and with the quality evals of the model-side course.

Explain it without notes

What is an eval, and what do the evals a frontend team owns check?

What is the difference between a scripted and a live eval?

Why is schema validity a good first eval?

How do you decide which eval failures block a merge?

Practice

Add a check "no more than 2 sentences" to the second example's checks. Which v1 replies fail it? Is that a good rule for the chat bubble?

Write a live-eval runner sketch: for each case call the model 3 times and report the pass rate per check. Show where the model would be created.

Trade-offs

↔
Strict checks catch more regressions and fail more often for harmless reasons. Start strict on what breaks the screen and loose on style.
↔
Live evals show real behaviour and cost money and time. Scripted ones are free and show only how your code handles what you imagined. Use both, with different schedules.

Done when you can

I can describe an eval case: input, reply source, checks, expected verdict.
I can validate a reply against the UI's schema and treat an invalid one as a handled outcome.
I can write checks for length, raw HTML, unsupported formatting and refusals.
I can compare two prompt versions on the same cases.
I can say which failures block a merge and which are tracked as rates.

Command Palette

Words you'll meet

Step by step

01Cases, checks, a verdict

02Two prompts, same cases

03Run them in CI

04When a check fails

Try it yourself

Add a case you saw in production

Soften a check

Make the eval fail the build

Code & diagrams

Break it on purpose

Accept any status string

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can