PHASE 11Advanced ~24 min· topic 4 of 8

Topic 11.4

Metrics That Matter

In one line

Metrics turn thousands of call records into a few numbers you can watch and alert on: latency percentiles, time to first token, tokens, cost, error and retry rates and cache hit rate.

Think of it like this

A car dashboard. You do not read a log of every engine revolution; you glance at speed, fuel and temperature. If one needle goes into the red you act. Metrics are the needles: a few numbers, always visible, computed from the many small events.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Percentile (p50, p95): The value below which that share of calls falls: p95 is slower than 95 of 100 calls.
Nearest-rank rule: Sort the values and take the one at position ceil(p x n / 100), counting from 1.
TTFT: Time to first token: how long before the first piece of a streamed reply arrives.
Error rate: Failed calls divided by all calls, usually as a percentage per time window.
Cache hit rate: The share of lookups answered from the cache.
Counter / gauge / histogram: Metric types: a counter only goes up, a gauge goes up and down, a histogram counts values into buckets.
Label: A named tag on a metric (model, feature, status) used to slice it; keep the set of values small.
Cardinality: How many distinct label combinations a metric has; very high cardinality makes metric stores slow and costly.

Step by step

01Step 1: start from records

Twenty call records, each with latency, time to first token, tokens in and out, status and cache result. One call timed out at 30 seconds and one failed with an error. Everything in this topic is computed from this list.

02Step 2: the percentile function

Sort the values, find the rank with integer arithmetic ((p * n + 99) // 100 is ceil(p x n / 100)), and take that position. Doing the ceiling with integers avoids float surprises, which is why Python, Java and JavaScript give the same answer.

percentile.pywhole filepython

def percentile(values, p):
    ordered = sorted(values)
    rank = (p * len(ordered) + 99) // 100
    return ordered[rank - 1]

print(percentile([620, 580, 710, 3100], 50))  # rank 2 -> 620

03Step 3: read the mean next to the percentiles

The mean is 2,325 ms, p50 is 650 ms and p95 is 3,100 ms. The mean describes nobody. p50 says the typical call is fine. p95 says 1 call in 20 is five times slower. The single 30-second call does not even show at p95, which is why you also chart the maximum or p99 and count timeouts.

Step 3: read the mean next to the percentilesdiagram

Rendering diagram…

04Step 4: measure a stream

For a stream, record when the first token arrived and when the last did. The example simulates starts-fast (first token at 300 ms, a token every 25 ms) and starts-slow (900 ms, a token every 12 ms). For 10 tokens the fast starter finishes first (525 ms against 1,008 ms). For 120 tokens the slow starter wins (2,328 ms against 3,275 ms).

05Step 5: pick the metric by the product

A chat window cares about TTFT. A batch summariser cares about total time and cost. A voice assistant cares about both. Choose the numbers on your dashboard from what your users feel, not from what the library prints by default.

06Step 6: record real metrics with a library

The code block below shows Prometheus-style counters and a latency histogram in Python and JavaScript, and the Micrometer equivalent in Java. Prometheus scrapes them and histogram_quantile estimates p95 from the buckets. Hectal's Observability course (/observability) covers Prometheus and Grafana.

Try it yourself

1
Predict p95 before running
For the 20 latencies, write down the sorted list's 19th value by hand, then check it against the output. Try p99: what rank does the rule give, and what does it show?
2
Add a model to the stream comparison
Add a third model balanced with first token at 500 ms and one token every 18 ms. For 10 and for 120 tokens, which model finishes first?
3
Use the labs
Run the first lab three times and note the spread in time to first token. Then run the second lab. Does a hidden-reasoning model start later? Does the number of output tokens explain the total time?

Code & diagrams

Call records to metrics New tab

Twenty scripted records. Percentiles use the nearest-rank rule. Prices are example numbers: check your provider.

Expected output

calls: 20, errors: 2 (10.0%)
latency mean 2325 ms, p50 650 ms, p95 3100 ms
time to first token p50 160 ms, p95 900 ms
tokens in 19016, out 2540, cost 0.013318 dollars
cache hit rate: 30.0%

Time to first token and total time New tab

Two simulated models. A short reply favours the quick starter, a long reply favours the fast decoder.

Expected output

--- a reply of 10 tokens
starts-fast: first token 300 ms, finished 525 ms, 40 tokens/s
starts-slow: first token 900 ms, finished 1008 ms, 83 tokens/s
--- a reply of 120 tokens
starts-fast: first token 300 ms, finished 3275 ms, 40 tokens/s
starts-slow: first token 900 ms, finished 2328 ms, 83 tokens/s

Recording the same metrics with real librariespython

Not run here. Python uses prometheus-client, JavaScript prom-client, Java Micrometer (which Prometheus can scrape). Metric names are examples.

# pip install prometheus-client
import time

from prometheus_client import Counter, Histogram, start_http_server

LLM_REQUESTS = Counter("llm_requests_total", "Model calls", ["model", "status"])
LLM_TOKENS = Counter("llm_tokens_total", "Tokens used", ["model", "direction"])
LLM_LATENCY = Histogram(
    "llm_request_duration_seconds", "Model call latency", ["model"],
    buckets=(0.25, 0.5, 1, 2.5, 5, 10, 30),
)

start_http_server(8000)   # Prometheus scrapes http://host:8000/metrics


def record_call(model, status, tokens_in, tokens_out, started):
    LLM_REQUESTS.labels(model=model, status=status).inc()
    LLM_TOKENS.labels(model=model, direction="input").inc(tokens_in)
    LLM_TOKENS.labels(model=model, direction="output").inc(tokens_out)
    LLM_LATENCY.labels(model=model).observe(time.perf_counter() - started)

PromQL: p95 latency from the histogrampromql

Prometheus estimates a percentile from histogram buckets, so the result is an estimate within the bucket widths.

histogram_quantile(0.95, sum by (le, model) (rate(llm_request_duration_seconds_bucket[5m])))

Streaming timing: a small reasoning modelreal model

Try: Send with streaming on. Under the reply read three numbers: time to first token, total time and token usage. Write them down, then use the next lab to repeat with a different model.

System

user

Modelsmall, fast, reasons firstTemperature · 0.2Max tokens (1–1024)Reasoning effort Stream (SSE) JSON mode

Goals

The reply finished on its own (finish_reason is stop)

Streaming timing: a mid-size model without hidden reasoningreal model

Try: Send the same request to a different model, then compare the three numbers with the previous lab: time to first token, total time and tokens used. Which model starts sooner? Which finishes sooner? Which used more output tokens?

System

user

Modelmid-size, also reads imagesTemperature · 0.2Max tokens (1–1024) Stream (SSE) JSON mode

Goals

The reply finished on its own (finish_reason is stop)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Take a percentile of unsorted values

In percentile, replace ordered = sorted(values) with ordered = list(values).

terminal

$ Run

── what you'll see ──

calls: 20, errors: 2 (10.0%)

latency mean 2325 ms, p50 630 ms, p95 615 ms

time to first token p50 155 ms, p95 165 ms

tokens in 19016, out 2540, cost 0.013318 dollars

cache hit rate: 30.0%

Myth vs fact

Myth

Average latency is a good summary.

Fact

A few very slow calls drag the mean up and a fast majority drags it down, so it describes nobody. Use p50 and p95 or p99.

Myth

Metrics need a big monitoring stack.

Fact

Start with a list of records and a function like the one above. A stack makes it continuous; the definitions are the hard part.

Myth

A lower time to first token means a faster answer.

Fact

It means the answer starts sooner. Total time also depends on output length and decoding speed.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Define each metric exactly and write it down: which calls count (retries too?), which unit (ms), which window. Two teams with the same metric name and different definitions will argue for hours.
▸
Percentiles cannot be averaged. Do not average p95 values from several servers. Keep histograms or the raw values and compute the percentile over the combined data.
▸
Add finish_reason counts and parse-failure counts as metrics. A reply cut off at max_tokens or in the wrong format returns HTTP 200 and still hurts users.

Remember this

1
Why this exists. One million call records cannot be read. Metrics summarise them over time (per minute, per hour) into numbers that are cheap to store, quick to chart and easy to alert on. Logs and traces tell you what happened to one request; metrics tell you whether the system as a whole is healthy, and when it changed.
2
Latency percentiles, not averages. The mean hides the slow calls. In the example, 20 calls have a mean latency of 2,325 ms because one timed out at 30,000 ms, yet the typical call takes about 650 ms. A percentile says what share of calls was at or below a value: p50 (the median) is the typical call, p95 means 95 of 100 calls were that fast or faster, so 1 in 20 is slower. Users feel the slow tail, so watch p95 and p99. The course uses the nearest-rank rule so all three languages agree: sort the values, take the one at position ceil(p x n / 100) counting from 1. For 20 values, p95 is the 19th.
3
Time to first token and total time. For streamed replies (Phase 5) there are two waits. Time to first token (TTFT) is the wait before anything appears, and decides whether the app feels alive. Total time is when the last token arrives. A model can start faster but finish later, so a short answer favours the fast starter and a long answer favours the fast decoder. The second example simulates two models and prints both numbers and a tokens-per-second speed. The lab below measures them for real.
4
Tokens and cost. Count input and output tokens separately (output usually costs more per token), per model and per feature, and multiply by a price table kept in configuration. Cost per request and cost per hour are the numbers that catch a prompt that suddenly got longer. Use the usage the provider returns, not your own guess, because tokenizers differ (Topic 1.2) and reasoning tokens are hidden (Topic 1.8). This site's AI tutor keeps a similar simple usage metric: calls per user per day in an ai_usage table, which enforces its daily limit.
5
Error rate and retry rate. The error rate is failed calls divided by all calls, split by cause (rate limit 429, timeout, overload 5xx, bad request 4xx, invalid output). The retry rate is how often calls needed a second try; a rising retry rate warns you before the error rate does (Topic 6.8). Also count finish_reason of length (cut off) and parse failures: they are errors the status code does not show.
6
Cache hit rate. If you cache replies or use the provider's prompt caching (Phase 6), the hit rate is hits divided by lookups. A drop from 40% to 5% after a deploy often means a prompt prefix changed and the cache stopped matching, which shows up as a cost and latency rise at the same time.
7
Labels and cost of metrics. Add labels (model, feature, status) so you can slice a metric, but keep the number of distinct label values small. A label such as user id creates one time series per user and can overwhelm your metric store. Put per-user detail in logs, not metric labels. Prometheus-style tools use three metric types: a counter only goes up (requests, tokens), a gauge goes up and down (in-flight requests), a histogram counts values into buckets so percentiles can be estimated from the buckets.

Explain it without notes

Why are percentiles better than the mean for latency, and what does p95 mean?

What is the difference between time to first token and total time, and when does each matter?

Which metrics would you put on a dashboard for an LLM feature, and why?

Practice

Write percentile(values, p) with the nearest-rank rule for 1 <= p <= 100 and print p50, p90 and p100 of [7, 3, 9, 1, 5, 8, 2, 6, 4, 10].

Write cache_hit_rate(records) where each record is "hit" or "miss"; return the percentage with one decimal as a string such as "33.3%", and "n/a" for an empty list.

Trade-offs

↔
Fine-grained labels let you slice metrics and make them costly; keep labels to model, feature and status.
↔
Short windows react fast and are noisy; long windows are stable and slow to notice a problem.
↔
Histogram buckets are cheap and give estimated percentiles; raw values give exact ones and cost storage.

Done when you can

Done when you can compute p50 and p95 with the nearest-rank rule by hand.
Done when you can explain why the mean misleads and what to chart instead.
Done when you can measure time to first token and total time for a stream.
Done when you can list the metrics an LLM feature needs and the label values to keep small.
Done when you can record a counter and a histogram in one real library.

Command Palette

Words you'll meet

Step by step

01Step 1: start from records

02Step 2: the percentile function

03Step 3: read the mean next to the percentiles

04Step 4: measure a stream

05Step 5: pick the metric by the product

06Step 6: record real metrics with a library

Try it yourself

Predict p95 before running

Add a model to the stream comparison

Use the labs

Code & diagrams

Break it on purpose

Take a percentile of unsorted values

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can