Topic 11.1
Why LLM Apps Need Observability
In one line
You cannot debug what you did not record. LLM calls add three problems to ordinary services: every call costs money, the same input can give different outputs, and one call can take many seconds.
Think of it like this
Driving a car with no dashboard and no black box. There is no speedometer, no fuel gauge, no warning light, and no record of the trip. The car runs fine until it does not, and then nobody can say how fast you were going, how much fuel is left, or what happened in the last minute. Observability is the dashboard (what is happening now) plus the black box (what happened then) for your software.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Observability
- How well you can understand what a system did from the records it left behind (logs, traces, metrics).
- Log
- A record of one event, such as one model call, with its details.
- Trace
- The path of one request through all its steps, with the time each step took.
- Span
- One step inside a trace, with a start time, an end time and a parent step.
- Metric
- A number measured over time, such as calls per minute or p95 latency.
- Latency
- How long a call takes from sending the request until the reply is complete.
- Non-determinism
- The same input can give different outputs, so you cannot reproduce a bad answer by re-running it.
- Feature
- One product function that calls a model, such as search help or summarising; costs and errors are tracked per feature.
Step by step
01Step 1: the blind app
The app serves ten requests. All it can say afterwards is "10 requests served". It cannot say what they cost, how long they took, or which failed, because nothing was written down. This is the starting point of most first versions.
02Step 2: write one record per call
After every model call, write one record: feature, model, tokens in and out, latency, status. This is a hand-made log. It is a few lines of code and it changes what you can answer.
calls.append(("summarise", "large", 3200, 400, 2900, "ok"))
# feature, model, tokens_in, tokens_out, latency_ms, status03Step 3: ask questions of the records
With the records, grouping is easy: cost per feature (tokens times price), failures per feature, the slowest call. The example prints exactly those. The answer to "why is the bill high?" is now one line: the summarise feature sends about 3,000 input tokens per call.
04Step 4: notice what the records do not say
The timeout row shows 30,000 ms and no output tokens. The records say that it failed, not why: was the prompt too long, the provider slow, or the network down? Good records have an error type, not only a status. Each later topic adds one missing piece.
05Step 5: the rest of the phase
Topic 11.2 makes the logs safe. Topic 11.3 adds traces. Topic 11.4 turns records into metrics. Topic 11.5 alerts on them. Topic 11.6 adds what users say. Topic 11.7 practises incidents. Topic 11.8 wraps it into one client you can copy.
Try it yourself
- 1
Predict the most expensive feature
Before running, estimate which feature costs the most using only the token counts (input costs 0.50 and output 1.50 dollars per million tokens). Then run and compare with the output.
- 2
Add a question
Add a line that prints the average latency per feature. Which feature is slowest on average, and does that match the feature that costs the most?
- 3
Switch languages
Run the example in all three languages. Find where each language groups records by feature (a dict, an object, a
LinkedHashMap).
Code & diagrams
Ten scripted calls. The blind view can only count them. The records answer cost, failures and the slowest call. Prices are example numbers.
Expected output
blind view: 10 requests served
search-help: 3 calls, 0 failed, cost 0.000999 dollars
summarise: 3 calls, 1 failed, cost 0.005970 dollars
chat: 4 calls, 1 failed, cost 0.002795 dollars
slowest call: summarise took 30000 ms and ended as timeout
which feature costs most: summariseBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count only the failure you remembered
In the Python version, change row["bad"] += status != "ok" to row["bad"] += status == "error", so only the status error counts as failed.
Myth vs fact
Myth
Logging is something you add after launch.
Fact
The first bad answer or surprise bill arrives within days. Records you did not write cannot be recovered afterwards.
Myth
The provider's dashboard tells me everything.
Fact
It shows your totals, not which of your features, prompts or users caused them, and it knows nothing about wrong answers.
Myth
Observability means a big tool.
Fact
It starts with one structured line per call. Tools help you store and search the lines; the habit comes first.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Put a request id on every record and pass it through every step and service. Joining logs, traces and user feedback by one id is what makes the three signals useful together.
- ▸
Record the prompt version, model name and settings (temperature, max tokens) on every call. When quality changes, these are the first things to compare, and a re-run cannot recover them (Topic 3.8).
- ▸
Budget your observability: a trace and a log per call at a million calls a day is real storage. Sample healthy calls, keep every error, and aggregate early into metrics (Topics 11.2 and 11.4).
Remember this
- 1
Why this exists. Topic 0.4 showed that LLM features fail quietly: the call succeeds, the text is wrong, and nothing crashes. A crash at least leaves a stack trace. A quiet failure leaves nothing unless you recorded something. Without records, a user report like "it was slow and wrong on Tuesday" cannot be answered, a bill that doubled cannot be explained, and a prompt change cannot be proven good or bad.
- 2
What makes LLM calls different from ordinary calls. First, cost: a database query costs almost nothing, but a model call costs money in proportion to its tokens. So money becomes something you must measure per feature and per user. Second, non-determinism: the same input can produce different outputs. To investigate a bad answer you must have recorded the exact input, the model, the settings and the prompt version, because running it again will not give the same thing. Third, latency: a call takes hundreds of milliseconds to many seconds, and with streaming there are two times that matter, the wait for the first token and the wait for the last.
- 3
Three kinds of signal. Logs are events with details ("request r7 used the small model, 900 tokens in, failed with 503"). Traces show the path of one request through its steps with the time each step took (retrieval 45 ms, model call 620 ms, tool 180 ms). Metrics are numbers over time (calls per minute, p95 latency, dollars per hour). Logs answer "what exactly happened to this request?", traces answer "where did the time go?", metrics answer "is something wrong right now?". You need all three, and the rest of the phase builds each one.
- 4
Decide what to record by the questions you will ask. Before adding a field, write down the question it answers at 3 a.m. "Why did the bill double?" needs tokens and feature per call. "Which prompt version did this user get?" needs the prompt version on every record. "Is the provider down or is it us?" needs status and error type per call. Fields nobody will ask about only add cost and risk.
- 5
Monitoring versus observability. Monitoring watches for failures you already know about (error rate above 5%). Observability means your records are rich enough to ask a question you did not think of in advance ("did the failures start after the 14:05 deploy, and only for the summarise feature?"). LLM apps need the second kind, because their failure modes are new and strange.
- 6
It is not free. Records cost storage and money, and logs of prompts can contain personal data. Topic 11.2 covers logging safely, and sampling keeps the volume sane. The example below uses ten recorded calls to answer three questions the blind version cannot: which feature costs the most, which calls failed, and what the slowest call was.
Explain it without notes
Name three ways an LLM call differs from an ordinary service call, and what each means for what you record.
What question does each of logs, traces and metrics answer best?
Why can you not simply re-run a failed LLM request to see what went wrong?
Practice
Write cost_by_feature(calls) for tuples (feature, tokens_in, tokens_out) that returns a dict of feature to dollars, using example prices of 0.50 and 1.50 dollars per million tokens.
Write slowest_feature(calls) for tuples (feature, latency_ms) that returns the feature with the highest average latency (whole milliseconds, ties go to the feature that appears first).
Trade-offs
- ↔
More fields per record answer more questions but cost storage and raise privacy risk; record what a real question needs.
- ↔
Recording everything for every call gives perfect detail at high cost; sampling healthy calls and keeping all errors is the usual compromise.
- ↔
Building the basics by hand teaches what the tools do, but production systems should use an established library and format (Topics 11.3 and 11.8).
Done when you can
Done when you can name what makes LLM calls different (cost, non-determinism, latency).
Done when you can say what logs, traces and metrics each answer.
Done when you can list the fields one call record needs, and the question behind each.
Done when you can explain why a re-run is not a reproduction of a bad answer.