Structured logs, then metrics read back from them
The clock is a counter. Scores are the shared-word counts of the toy retriever used throughout the course.
When an answer is wrong, you need to know what retrieval did: which passages came back, with what scores, and how long each stage took. Structured log lines and trace spans record that, and a few metrics tell you when to look.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write parse_logfmt(line) that splits a line of key=value pairs separated by spaces into a dict (the value may contain =). Print the outcome of ts=2 event=answer req=r3 outcome=refused.
Write slowest(spans) taking a list of (name, start, end) tuples and returning the name of the one that took longest (the first, on a tie). Try it on three spans.
Explain it without notes
Why does a RAG system need more than error logs?
What would you log for each retrieval, and what would you be careful not to log?
What is a span, and what do the gaps in a trace tell you?
Name four metrics you would alert on for a RAG service and what each can reveal.
Expected output
ts=1 event=retrieve req=r1 q_words=6 index=8 hits=D8:3,D10:2 top=3
ts=2 event=answer req=r1 outcome=answered prompt_words=47
ts=3 event=retrieve req=r2 q_words=7 index=8 hits=D2:5 top=5
ts=4 event=answer req=r2 outcome=answered prompt_words=27
ts=5 event=retrieve req=r3 q_words=10 index=8 hits=D1:2,D10:2 top=2
ts=6 event=answer req=r3 outcome=answered prompt_words=57
...
requests: 7
refusal rate: 28.6 percent
mean top score: 2.00
weak evidence (top <= 2): 5 of 7
ALERT: refusal rate above 25 percent