PHASE 11Advanced ~22 min· topic 5 of 8

Topic 11.5

Dashboards and Alerts

In one line

A dashboard shows health at a glance, and an alert wakes a person only when something needs action. Good LLM alerts watch error rate, latency, cost anomalies and quality signals over time windows, and avoid firing on noise.

Think of it like this

A smoke detector in a kitchen. If it beeped every time you toasted bread, you would take out the battery, and then it would be silent in a real fire. A good alert is a smoke detector tuned to fire for fires, not toast. A dashboard is the window you look through to see the kitchen yourself.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Dashboard: A screen of charts that shows the system's health at a glance.
Alert: An automatic message sent to a person when a rule is true.
Page: To alert a person urgently, for example by phone, because action is needed now.
Window: A fixed slice of time (like 5 minutes) over which a metric is computed.
Baseline: What is normal for a metric, used to spot an anomaly.
Alert fatigue: People ignoring alerts because too many are false alarms.
Runbook: A short written procedure for what to do when an alert fires.
Resolved: The alert condition is no longer true.

Step by step

01Step 1: start with windows

Traffic arrives as individual events with a time. The first job is to put each event into a 5-minute window (timestamp // WINDOW_MS) and total up requests, errors and cost per window. Every rule then reads these small totals, not raw events.

02Step 2: write each rule as a function of one window

The error-rate rule says: at least 10 requests and more than 25% errors. The cost rule says: cost above three times the baseline. A rule is a plain function that returns true or false for a window. That makes it trivial to test.

rules.pywhole filepython

rules = {
    "error-rate": lambda r, e, c: r >= MIN_REQUESTS and e * 100 // r > ERROR_RATE_LIMIT,
    "cost-anomaly": lambda r, e, c: c > 3 * BASELINE_COST,
}

03Step 3: require several windows in a row

Window 3 has 42% errors, but the rule must not fire yet: one bad window may be a blip. A streak counter goes up when the rule is true and resets to 0 when it is not. The alert fires when the streak reaches 2, so it fires at the end of window 4, not window 3.

Step 3: require several windows in a rowdiagram

Rendering diagram…

04Step 4: fire once, then resolve

Without a firing flag the rule would alert on every window while it stays bad. With the flag, the alert fires once when the streak reaches the limit and a "resolved" line prints when the rule goes false again. Real systems also repeat reminders at a slow rate, which is a separate setting.

05Step 5: tune against the history

With 1 window needed, the error alert fires one window earlier and also risks noise. With no minimum volume, a quiet night with 3 requests and 2 errors would page someone. Replay your rule over past data before deploying it: how many alerts would it have fired, and were they real?

06Step 6: write it for a real system

The Prometheus rule below expresses the same error-rate rule over the llm_requests_total counter from Topic 11.4. It uses rate over 5 minutes, a minimum traffic guard, and for: 10m.

Try it yourself

1
Change the streak
Set WINDOWS_IN_A_ROW to 3. Which windows does the error-rate alert fire in now? Does the cost alert still fire before traffic returns to normal?
2
Make a noisy night
Change window 2 to (3, 2, 400), which is 66% errors from only 3 requests. Why does MIN_REQUESTS stop an alert? Set it to 0 and rerun.
3
Add a rule
Add a third rule slow using a fourth value per window, the p95 latency in ms, alerting above 3,000. What would you need to store per window to compute it?

Code & diagrams

An alert rule over time windows New tab

Scripted traffic in 5-minute windows. Costs are in micro-dollars (1,000,000 = 1 dollar) so everything is a whole number. Both rules need 2 windows in a row.

Expected output

window 0: 20 requests, 5% errors, cost 8000
window 1: 22 requests, 4% errors, cost 9020
window 2: 20 requests, 10% errors, cost 7800
window 3: 21 requests, 42% errors, cost 8400
window 4: 20 requests, 50% errors, cost 8100
  ALERT error-rate: true for 2 windows in a row
window 5: 20 requests, 5% errors, cost 30000
  RESOLVED error-rate
window 6: 20 requests, 5% errors, cost 29600
  ALERT cost-anomaly: true for 2 windows in a row
window 7: 20 requests, 5% errors, cost 8000
  RESOLVED cost-anomaly

A Prometheus alert rule for the error rateyaml

Not run here. Uses the llm_requests_total counter with a status label from Topic 11.4. Names and thresholds are examples.

groups:
  - name: llm-alerts
    rules:
      - alert: LlmHighErrorRate
        expr: |
          sum(rate(llm_requests_total{status!="ok"}[5m]))
            / sum(rate(llm_requests_total[5m])) > 0.25
          and sum(rate(llm_requests_total[5m])) > 0.03
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "More than 25% of model calls are failing"
          runbook: "Topic 11.7 provider outage runbook"

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Alert on a single window and ignore volume

Set WINDOWS_IN_A_ROW = 1 and MIN_REQUESTS = 0.

terminal

$ Run

── what you'll see ──

window 0: 20 requests, 5% errors, cost 8000

window 1: 22 requests, 4% errors, cost 9020

window 2: 20 requests, 10% errors, cost 7800

window 3: 21 requests, 42% errors, cost 8400

ALERT error-rate: true for 1 windows in a row

window 4: 20 requests, 50% errors, cost 8100

window 5: 20 requests, 5% errors, cost 30000

RESOLVED error-rate

ALERT cost-anomaly: true for 1 windows in a row

window 6: 20 requests, 5% errors, cost 29600

window 7: 20 requests, 5% errors, cost 8000

RESOLVED cost-anomaly

Myth vs fact

Myth

More alerts mean safer systems.

Fact

Alerts people ignore make systems less safe. Few alerts that need action beat many that do not.

Myth

A dashboard replaces alerts.

Fact

Nobody watches it at 3 a.m. A dashboard explains; an alert interrupts.

Myth

Cost is a finance matter, not an on-call matter.

Fact

A runaway loop can spend a month's budget in hours. Cost anomalies need the same alerting as errors.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Alert on symptoms users feel (errors, slowness, cost rate), and chart causes (cache rate, retries) to diagnose. Alerting on every cause creates noise and duplicates.
▸
Use a hard cost cap in the client as well as an alert. An alert needs a person to react in time; a cap works at 3 a.m. with nobody awake (Topic 6.2).
▸
Add the deploy or config change to the alert message when possible: "started 4 minutes after prompt v9 went live" saves the first half hour of any incident.

Remember this

1
Why this exists. Metrics (Topic 11.4) only help if someone sees them at the right moment. Nobody watches a chart all day, so you need two things: a dashboard that a person opens to understand the system, and alerts that interrupt a person only when action is needed. LLM apps need alerts for failures that look healthy to ordinary monitoring, above all runaway cost.
2
What to chart. Put the same few panels on one screen: requests per minute, error rate by cause, p50 and p95 latency (and TTFT for streams), tokens per request and cost per hour. Add the cache hit rate and the share of replies with finish_reason=length or a parse failure. Add deploy markers (a line when a new prompt or model went live), because most changes in these charts start at one. Hectal's Observability course (/observability) shows how to build such dashboards.
3
What to page on. Page (wake someone up) only for things that need a human now and are caused by you or the provider. Examples are a high error rate over several minutes, latency far above normal, or a cost rate that would exhaust the budget within hours. Send everything else to a ticket or a chat channel: a slowly rising average prompt length, a falling cache hit rate, a dip in thumbs-up. The test: "if this fires at 3 a.m., is there something a person can do about it right now?" If not, it is not a page.
4
Rules over windows. A rule is a condition evaluated over a time window: error rate over the last 5 minutes, cost in the last hour. Two guards cut noise. First, a minimum volume: an error rate of 50% from 2 requests is meaningless, so require at least some number of requests. Second, for how long: fire only when the condition holds for N windows in a row (Prometheus calls this the for duration). The example evaluates two rules over 5-minute windows with both guards, prints when each fires and when it resolves.
5
Cost anomaly alerts. A cost spike is the classic LLM incident: a prompt that grows, a loop that retries, a feature used a hundred times more than planned, a cache that stopped matching. Compare the current window with a baseline: here a window that costs more than three times the usual fires. A fixed budget alert is a second safety net: alert at 50% and 80% of the daily budget, and have a hard cap in the client itself (the budget guards from Topic 6.2).
6
Alert quality. Every alert needs an owner, a link to the dashboard, and a runbook step (Topic 11.7). Track how many alerts fire and how many needed action; if most do not, tune or delete them. Alert fatigue is a real failure mode: after a week of false alarms, people stop reacting to the real one.
7
A real alert rule. In Prometheus, a rule has an expression, a for duration, labels and annotations. The code block below shows an error-rate alert in that format. Hectal's SRE course (/sre) covers service-level objectives and alerting on the speed at which you spend an error budget, which is the more advanced version of what this topic builds by hand.

Explain it without notes

What is the difference between a dashboard and an alert, and what should wake someone up?

Why do alert rules use a minimum volume and a "for N windows" condition?

How would you detect a cost anomaly in an LLM app, and what would you put in place besides an alert?

Practice

Write fires(flags, n) where flags is a list of booleans (one per window) and the alert fires the first time n true values appear in a row. Return the 0-based index of that window, or -1 if it never fires.

Write cost_alert(costs, baseline, factor) returning the list of window indexes where the cost is above factor times baseline.

Trade-offs

↔
Longer "for" durations reduce false alarms and delay real ones.
↔
Tight thresholds catch small problems and page too often; loose thresholds are quiet and miss slow degradations.
↔
Alerting on cost in dollars is simple; alerting on tokens per request finds prompt growth earlier and is not affected by price changes.

Done when you can

Done when you can group events into windows and compute a rate per window.
Done when you can write a rule with a minimum volume and a streak requirement.
Done when you can say what to page on and what to send to a ticket.
Done when you can set a cost anomaly alert and a hard cap.

Command Palette

Words you'll meet

Step by step

01Step 1: start with windows

02Step 2: write each rule as a function of one window

03Step 3: require several windows in a row

04Step 4: fire once, then resolve

05Step 5: tune against the history

06Step 6: write it for a real system

Try it yourself

Change the streak

Make a noisy night

Add a rule

Code & diagrams

Break it on purpose

Alert on a single window and ignore volume

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can