Blind app versus recorded calls
Ten scripted calls. The blind view can only count them. The records answer cost, failures and the slowest call. Prices are example numbers.
You cannot debug what you did not record. LLM calls add three problems to ordinary services: every call costs money, the same input can give different outputs, and one call can take many seconds.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write cost_by_feature(calls) for tuples (feature, tokens_in, tokens_out) that returns a dict of feature to dollars, using example prices of 0.50 and 1.50 dollars per million tokens.
Write slowest_feature(calls) for tuples (feature, latency_ms) that returns the feature with the highest average latency (whole milliseconds, ties go to the feature that appears first).
Explain it without notes
Name three ways an LLM call differs from an ordinary service call, and what each means for what you record.
What question does each of logs, traces and metrics answer best?
Why can you not simply re-run a failed LLM request to see what went wrong?
Expected output
blind view: 10 requests served
search-help: 3 calls, 0 failed, cost 0.000999 dollars
summarise: 3 calls, 1 failed, cost 0.005970 dollars
chat: 4 calls, 1 failed, cost 0.002795 dollars
slowest call: summarise took 30000 ms and ended as timeout
which feature costs most: summarise