An exact-match cache with a TTL and safety rules
The clock is simulated. Normalising the prompt makes the second request a hit even though it has extra spaces.
An exact-match cache stores the answer to a request and returns it when the same request comes again, skipping the model call completely. It is free, instant and safe only when the answer should not change, so it needs a careful key, an expiry time and clear rules for what never to cache.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write cache_key(model, prompt, temperature) that returns the model, the temperature to one decimal and the prompt with spaces collapsed, joined by |. Print the key for the model "small", the prompt " Hello world " (with extra spaces) and the temperature 0.
Write TtlCache(ttl_ms) with put(key, value, now) and get(key, now) returning None when missing or expired. Show a value found at t=500 and gone at t=1500 with a TTL of 1000.
Explain it without notes
What must go into the cache key for a model call, and why?
When should you not cache a model answer? Give three cases.
What does TTL do, and how do you choose it?
What is a cache stampede and how do you reduce it?
Expected output
t=0: MISS What is DNS? (temperature 0.0, personal no)
t=5000: HIT What is DNS? (temperature 0.0, personal no)
t=6000: MISS What is DNS? (temperature 0.8, personal no)
t=7000: MISS What is DNS? (temperature 0.8, personal no)
t=8000: MISS What is my balance? (temperature 0.0, personal yes)
t=9000: MISS What is my balance? (temperature 0.0, personal yes)
t=70000: MISS What is DNS? (temperature 0.0, personal no)
t=71000: HIT What is DNS? (temperature 0.0, personal no)
model calls: 6 for 8 requests