Command Palette

Search for a command to run...

Hectal
Phase 11Advanced12 of 15 in LLM Engineering

Observability and Operations

See what your LLM app is doing: safe logs, traces, metrics, alerts, user feedback and incident drills, built by hand and then with real tools.

A model call that works in a demo teaches you nothing about production. In production the bill doubles overnight. A prompt change quietly makes answers worse. One provider has a bad hour. A user says "it gave me a wrong answer yesterday" and you cannot find the call. This phase is about not being blind: recording enough about every call to answer those questions, without recording things you must not keep.

You will build the three signals by hand on a simulated clock, with fake models and scripted traffic, so every number is repeatable. The signals are structured logs with a redactor and a sampler, trace spans for retrieval, model calls and tools, and metrics such as p50 and p95 latency, time to first token, tokens, cost and cache hit rate. Then you will write an alert rule over time windows and join user feedback to requests. Last, you will run incident drills where your own code detects the problem.

Real tooling appears next to the hand-built versions: OpenTelemetry spans in Python, Java and JavaScript, the OpenTelemetry generative-AI attribute names (and their current status), Prometheus-style metrics and an alert rule. The phase ends with an instrumented model client in all three languages. The ideas link to Hectal's Observability course (/observability) and SRE course (/sre), which go deeper on the general tools.

0/8 · 0%
8 topics ~3 h 20 code blocks & diagrams
Start with the first topic
1
11.1

Why LLM Apps Need Observability

You cannot debug what you did not record. LLM calls add three problems to ordinary services: every call costs money, the same input can give different outputs, and one call can take many seconds.

20 min 1 code practice

2
11.2

Logging Prompts and Responses Safely

Prompts and replies are the most useful thing to log and the most dangerous: they contain names, emails, secrets and private documents. Safe logging uses structured lines, a redactor before the write, sampling, and a retention limit.

20 min 2 code practice

3
11.3

Tracing LLM Calls

A trace shows one request as a tree of timed steps: retrieval, model calls, tools. It answers where the time went and links the services a request crosses. OpenTelemetry is the standard way to produce traces.

23 min 3 code practice

4
11.4

Metrics That Matter

Metrics turn thousands of call records into a few numbers you can watch and alert on: latency percentiles, time to first token, tokens, cost, error and retry rates and cache hit rate.

24 min 6 code practice

5
11.5

Dashboards and Alerts

A dashboard shows health at a glance, and an alert wakes a person only when something needs action. Good LLM alerts watch error rate, latency, cost anomalies and quality signals over time windows, and avoid firing on noise.

22 min 2 code practice

6
11.6

User Feedback Loops

Thumbs up and down, with a reason, tell you what automatic metrics cannot: whether the answer was useful. Join each vote to the request that produced it, compare prompt versions, and send the bad answers into your golden set.

20 min 1 code practice

7
11.7

Incident Drills

An incident is when the system is failing users right now. LLM apps have four typical ones: provider outage, cost spike, prompt regression and an injection report. A runbook and a practised drill turn panic into a checklist.

22 min 2 code practice

8
11.8

Playbook: An Instrumented Model Client

One wrapper around any model that leaves a redacted JSON log line, a trace span and a metrics record for every call, plus a small summary that checks the metrics against limits. Copy it and plug in your real client.

22 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Redact, then log

Write redact(text) that replaces every email (letters, digits, dots, dashes and underscores, then @, a domain with a dot) with [EMAIL] and every run of exactly 10 digits (not part of a longer number) with [PHONE].

Intermediate
2

Nearest-rank percentiles and a latency report

Write percentile(values, p) using the nearest-rank rule: sort the values and return the one at position ceil(p * n / 100) counting from 1. Use integer arithmetic only: (p * n + 99) // 100.

Intermediate
3

Print a trace tree with self time

Spans are tuples (id, parent, name, start_ms, end_ms) in start order; a parent of 0 means the root. Write span_tree(spans) that returns a list of lines, one per span, indented two spaces per level of depth.

Intermediate
4

An alert rule with a minimum volume

windows is a list of (requests, errors) per window. Write error_alert(windows, limit_percent, min_requests, in_a_row) that returns the 0-based index of the window where the alert fires, or -1.

Advanced
5

Record the failure, then re-raise

ProviderError and a fake model are given. Write observed_call(model, prompt, records) that calls model.complete(prompt), which returns a dict with tokens_in, tokens_out and latency_ms, or raises ProviderError.

Advanced