Time to first token and total time
Two simulated models. A short reply favours the quick starter, a long reply favours the fast decoder.
Metrics turn thousands of call records into a few numbers you can watch and alert on: latency percentiles, time to first token, tokens, cost, error and retry rates and cache hit rate.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write percentile(values, p) with the nearest-rank rule for 1 <= p <= 100 and print p50, p90 and p100 of [7, 3, 9, 1, 5, 8, 2, 6, 4, 10].
Write cache_hit_rate(records) where each record is "hit" or "miss"; return the percentage with one decimal as a string such as "33.3%", and "n/a" for an empty list.
Explain it without notes
Why are percentiles better than the mean for latency, and what does p95 mean?
What is the difference between time to first token and total time, and when does each matter?
Which metrics would you put on a dashboard for an LLM feature, and why?
Expected output
--- a reply of 10 tokens
starts-fast: first token 300 ms, finished 525 ms, 40 tokens/s
starts-slow: first token 900 ms, finished 1008 ms, 83 tokens/s
--- a reply of 120 tokens
starts-fast: first token 300 ms, finished 3275 ms, 40 tokens/s
starts-slow: first token 900 ms, finished 2328 ms, 83 tokens/s