PHASE 13Advanced ~30 min· topic 5 of 8

Topic 13.5

Measuring an AI Feature

In one line

You cannot improve what you cannot see. Measure time to first token, speed, errors, stops and feedback as counts and buckets, and keep the text people typed and the answers they got out of your logs.

Think of it like this

A bus company that counts passengers and delays but does not record who sat where or what they said. The numbers show which route is late and which is empty, and nobody's private conversation ends up in a file. Measure how the service behaves, and leave the people's words out.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Time to first token (TTFT): How long from the click until the first word is on screen.
Tokens per second: How fast the answer streams: output tokens divided by the time between the first and last token.
Bucket: A named range such as fast, ok or slow. You log the bucket instead of the raw number.
Metric: A number you count or measure over time: requests, errors, stops.
Allowlist: A list of what is allowed. Anything not on it is dropped. The safe opposite of "log everything except…".
PII: Personally identifiable information: names, emails, addresses, order numbers, anything that points to a person.
Scrub: Replace sensitive patterns in text with placeholders such as [email] before logging.
Telemetry: Automatic records of what a system did, sent to a monitoring tool.

Step by step

01Measure in the browser

The first example wraps useChat. ask() stores performance.now() as start and remembers how many messages exist (base). An effect watches messages: when a message added after the click has text on screen, it stores first. onFinish stores the end time, divides the output tokens (from the metadata, Topic 13.1) by the seconds since first, and logs buckets.

The base check matters: without it the effect sees the previous answer's text right after the click and records a first token at 0 ms. See the break-it.

Measure in the browserdiagram

Rendering diagram…

02Three scripted behaviours

The stand-in server answers the first question with a fast start and a fast stream, the second with a slow start (400 ms before the first token) and the third with a fast start and a very slow stream. The log reads: fast / steady, ok / steady, fast / very slow. The numbers themselves are never printed.

03Count the outcomes

The second example sends six requests: two normal answers, one 429, one broken stream, one answer the user stops after the third word, and one more normal answer. The stop is triggered by the stream itself (an effect that sees three words on screen), never by a timer, so it stops after exactly three words on every machine.

onFinish fires for all three endings and tells you which with isAbort and isError. The page counts answered: 3, failed: 2, stopped: 1, plus three feedback clicks, and prints shares. Both the 429 and the broken stream count as failed.

04An allowlist for events

The third example is the logger. Each event name has a list of allowed keys, and each key has a list of allowed values (or the type number). A teammate adds question, email and answerText to an event and the log shows only ttft, speed and outcome, plus the note that three keys were dropped. An unknown event name is dropped entirely.

The last two lines scrub an error message. Emails, order ids and long numbers become placeholders. A street address passes straight through, which is why the better answer is to never log the message at all.

Try it yourself

1
Move a threshold
In the first example change bucketTtft so "fast" means under 20 ms. The first and third questions now log ok too, because their scripted start is about 40 ms. Thresholds decide what your dashboard calls bad, so write them down and agree them with the team.
2
Add a new event, safely
In the third example add stop: { after: "number" } to SCHEMA and call track("stop", { after: 3, text: "hello" }). The number is kept and the text is dropped. New events need a line in the schema, which is the moment to ask whether they are safe.
3
Break the scrubber
Call scrub("asha dot k at example dot com"). Nothing changes: the pattern needs a real email. People write sensitive things in ways no pattern expects.

Code & diagrams

Time to first token and speed, logged as buckets only

The delays are scripted, so the buckets repeat. The thresholds (250 ms, 1000 ms, 20 and 8 tokens per second) are made up for the example: choose yours from your own measurements.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

track answer {"ttft":"fast","speed":"steady"}
track answer {"ttft":"ok","speed":"steady"}
track answer {"ttft":"fast","speed":"very slow"}

Outcomes and feedback as counters, then as rates

Both the 429 and the broken stream end as failed. A real dashboard would split them by status code (a number is safe to log) so you can see a provider problem apart from your own bug.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

counts: {"answered":3,"failed":2,"stopped":1,"thumbs_up":2,"thumbs_down":1}
requests: 6 | answered: 50% | failed: 33% | stopped early: 17%
feedback: up 2 down 1 | thumbs-down share: 33%

An event logger that can only say what you allow, and what scrubbing misses

The example addresses and ids are made up. The allowlist is the real protection. The scrub function is only a second line of defence.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

track answer {"ttft":"fast","speed":"steady","outcome":"answered"}
track answer {"ttft":"ok","speed":"slow","outcome":"answered"} | dropped: question, email, answerText
track error {"status":429} | dropped: message
track feedback {"value":"down"} | dropped: comment
track dropped: unknown event "page_view"
Rate limited for [email] on [order], phone [number]
Lake Road, flat 12, Pune ← a street address passes straight through: scrubbing alone is not enough

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Count the previous answer as the first token

In the first example change the hasText line to const hasText = last && last.role === "assistant" && last.parts.some((p) => p.type === "text" && p.text); (remove the messages.length >= t.current.base + 2 && part).

terminal

$ Console, second line

── what you'll see ──

track answer {"ttft":"fast","speed":"slow"}

Myth vs fact

Myth

I need the exact question text to debug.

Fact

Usually you need the outcome, status, bucket and a request id. If you must look at content, do it in a protected place with a short retention time and the user's consent.

Myth

Logging the raw milliseconds is more useful.

Fact

Raw timings from many devices are noisy and identify devices. Buckets and server-side percentiles tell you what you need.

Myth

Scrubbing makes logging text safe.

Fact

Patterns catch what you thought of. Names, addresses and free text slip through. An allowlist avoids the question.

Myth

A high stop rate means users are impatient.

Fact

It may mean the first words are wrong or off topic, or the answer is too long. Compare with time to first token and the thumbs-down share before deciding.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Use percentiles for time (the slowest 5%), not averages. Averages hide the bad tail where the complaints come from. Compute them on the server side from timings you store as plain numbers next to a request id (no user text), or approximate them from bucket counts. Buckets alone cannot give an exact percentile.
▸
Send events with navigator.sendBeacon or a similar fire-and-forget call so a closing tab does not lose them, and never block the UI on analytics. Message metadata and usage on the SDK side are Topic 5.7.
▸
Give every request an id that appears in the page's event, the server log and the provider call. When a user reports a problem you can find the exact request without reading their text.
▸
Sample: log every error and a small share of successes. Error rates need every error. Speed buckets do not need every request.

Remember this

1
The problem this solves: An AI feature can fail quietly. The server returns 200, the page shows text, and the text is slow, wrong, or cut off. Without measurements you learn about it from an angry email. With careless measurements (logging every question and answer) you build a pile of private data that you must now protect, delete on request and explain to regulators. This topic shows what to measure and how to stay safe.
2
Time to first token (TTFT) is the time from the click to the first word on screen. It is what the user *feels* as waiting. Measure it in the browser with performance.now(): take a timestamp when the request starts, take another when the first text is on screen, subtract. Speed (tokens per second) is the output token count divided by the time between the first and last token. A feature can have a fast start and a slow stream, or the reverse, and the fixes differ (provider region and queueing for start, model size for speed).
3
Log buckets, not raw numbers. Raw milliseconds depend on the device and the network, and they are noisy. Compare them with thresholds and log the bucket: fast, ok, slow. The first live example does this: it measures with performance.now() but sends only { ttft: "ok", speed: "steady" }. Buckets are stable enough to chart, easy to alert on, and make a poor fingerprint of a person. Choose thresholds from your own data, not from this course.
4
Count outcomes. Every request ends one of a few ways: answered, failed, stopped by the user (a high rate means answers are slow or unwanted), or cut off (finish reason length). In useChat, onFinish receives isAbort, isDisconnect (a network error), isError and finishReason. Count each outcome, and count a disconnect on its own: it points at the user's network, not at your server. The second example turns the counts into rates: 50% answered, 33% failed, 17% stopped. A rising failed or stopped share is the first sign of trouble.
5
Count feedback. Thumbs up and down are the only direct signal of answer quality. Log the click (up or down), never the free-text comment unless you have a plan to protect it. The thumbs-down share per feature, per prompt version and per model is how you notice that a change made things worse.
6
What never to log: the user's text, the model's answers, email addresses and names, order numbers, addresses, API keys, and full error messages (provider errors may echo the prompt). The third example shows an allowlist: the logger accepts only known events with known keys and known values, and drops everything else, so a well-meaning teammate who adds question "to debug later" changes nothing. Where you must keep a message, scrub it first. Scrubbing patterns catch emails and ids and miss street addresses, so treat them as a safety net and not as permission.
7
Tracing and telemetry. The AI SDK has a telemetry option and integrations that record calls, steps and token usage for tools such as OpenTelemetry. They can include prompts and outputs: check each option's input/output recording setting and turn off what you should not store. Metrics (counts, buckets) are cheap and safe; traces with content are for a protected environment, with a retention limit.

Explain it without notes

How do you measure time to first token in the browser, and what is the common bug?

Why log buckets instead of raw numbers?

Which outcomes of a chat request should you count, and what does onFinish tell you?

What must never be in the logs of an AI feature, and how do you enforce that?

Practice

Extend the third example's schema with a fallback event that records status (a number) and mode (backup or main), and call it as the fallback in Topic 13.3 would.

In the second example add a counter length for answers whose finishReason is length, using the finishReason argument of onFinish.

Trade-offs

↔
More measurement helps you find problems and costs privacy, storage and attention. Start with a handful of counters and buckets and add a metric only when someone has a question it answers.
↔
Client-side timing shows what the user felt, including the network, and is easy to tamper with. Server-side timing is trustworthy and misses the network. Use both for different questions.

Done when you can

I can measure time to first token and speed with performance.now() and turn them into buckets.
I can count answered, failed, stopped and cut-off requests from onFinish.
I can build an allowlist logger that drops unknown events, keys and values.
I can list what must never be logged and say what scrubbing cannot catch.
I can give a request an id that connects the page, the server and the provider without containing user text.

Command Palette

Words you'll meet

Step by step

01Measure in the browser

02Three scripted behaviours

03Count the outcomes

04An allowlist for events

Try it yourself

Move a threshold

Add a new event, safely

Break the scrubber

Code & diagrams

Break it on purpose

Count the previous answer as the first token

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can