Topic 6.4
Exact-Match Caching
In one line
An exact-match cache stores the answer to a request and returns it when the same request comes again, skipping the model call completely. It is free, instant and safe only when the answer should not change, so it needs a careful key, an expiry time and clear rules for what never to cache.
Think of it like this
A teacher who gets the same question from twenty students writes the answer on the board once. Later students read the board instead of asking again. The board stays correct as long as the facts don't change, so at the end of the week the teacher wipes it and answers fresh. And if a student asks about their own marks, the teacher never writes that on the board.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Cache
- A place that remembers earlier answers so repeats can be served without doing the work again.
- Cache key
- The text that identifies a request. Equal keys mean "same question".
- Hit and miss
- A hit finds a saved answer. A miss finds nothing and must call the model.
- TTL (time to live)
- How long a saved answer is allowed to be used before it expires.
- Normalise
- Turn harmless differences (extra spaces) into one standard form so equal requests get equal keys.
- Hit rate
- Hits divided by all requests. It tells you how much a cache saves.
- Eviction
- Removing old entries when the cache is full.
Step by step
01Step 1: spot the repeat
Log your prompts for a day and count repeats. If a few prompts account for a large share of calls (FAQ questions, fixed classification prompts), a cache will pay for itself at once. If almost every prompt is unique, exact caching will not help.
02Step 2: build the key from everything that matters
The key joins model name, temperature, max tokens and the normalised prompt. If you later add a system prompt or tools to the request, they go in the key too. When the code builds the request, build the key from the same object so they can't drift apart.
def make_key(model_name, prompt, temperature, max_tokens):
return f"{model_name}|{temperature:.1f}|{max_tokens}|" + " ".join(prompt.split())03Step 3: store with an expiry
Each entry records when it stops being valid: now plus the TTL. A lookup that finds an expired entry counts as a miss. The example's clock is a number you advance yourself, so expiry is easy to test.
04Step 4: decide what is cacheable
Only deterministic, non-personal requests are saved. The decision happens before the lookup and before the save, so personal or random answers never enter the cache. In the output you can see those requests going to the model every time.
05Step 5: watch the TTL work
The last two requests come after the first entry's 60 seconds are over: the first misses (expired, so the model is called and the answer is saved again) and the second hits. Count the model calls at the end: 6 for 8 requests.
06Step 6: add the production details
Put a prompt version in the key so a prompt edit never serves old answers. Use a shared store with TTL support. Limit the size. Count hits and misses. And never cache an error or a cut-off reply (finish_reason: length): save only complete, valid answers.
Try it yourself
- 1
Predict the hits
Before running, mark which of the eight requests are hits. Then run and compare. Which rule stopped each miss from being saved?
- 2
Change the TTL
Set
TTL_MSto 10_000. Which requests change from hit to miss? Set it to 100_000 and check the t=70000 request. - 3
Include the user
Make the personal request cacheable by adding a
userargument to the key (for exampleana), and ask once asanaand once asbob. Does bob get ana's answer? Why not?
Code & diagrams
The clock is simulated. Normalising the prompt makes the second request a hit even though it has extra spaces.
Expected output
t=0: MISS What is DNS? (temperature 0.0, personal no)
t=5000: HIT What is DNS? (temperature 0.0, personal no)
t=6000: MISS What is DNS? (temperature 0.8, personal no)
t=7000: MISS What is DNS? (temperature 0.8, personal no)
t=8000: MISS What is my balance? (temperature 0.0, personal yes)
t=9000: MISS What is my balance? (temperature 0.0, personal yes)
t=70000: MISS What is DNS? (temperature 0.0, personal no)
t=71000: HIT What is DNS? (temperature 0.0, personal no)
model calls: 6 for 8 requestsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Leave the temperature out of the key and cache everything
In ask, change the cacheable test to cacheable = not personal and remove {temperature:.1f}| from the key (in every language). Run it.
Myth vs fact
Myth
A cache is always safe because it only returns what the model said.
Fact
It can return an answer made for a different user, a different question, or a time when it was true. The key and the rules decide whether that is a bug.
Myth
The longer the TTL, the better.
Fact
A long TTL saves more calls and serves more stale answers. Choose it from how fast the truth changes.
Myth
Caching changes nothing for testing.
Fact
It makes tests and demos stable and free, but you must clear it when a prompt changes, or you'll test yesterday's prompt.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cache stampede: when a popular entry expires, many requests miss at once and all call the model. Use a lock so one request refills while others wait, or refresh slightly before expiry.
- ▸
Add a prompt version and the model name to the key. A cache that survives a prompt change makes every evaluation (Phase 9) meaningless.
- ▸
Cache hits should still pass your output checks if the checks changed. Better: store only validated answers (Topic 4.8).
- ▸
Hash the key for storage (the key can be long), but log the readable form of the key when debugging a wrong hit.
- ▸
Temperature 0 makes replies very stable but does not promise identical text on every call, because providers can change models or hardware behind a name. A cache turns this into an advantage for users: they see one consistent answer until the TTL ends. Version the model in the key and expect the answer to change when the model does.
Remember this
- 1
Why this exists. Many products send the same request again and again: a popular FAQ question, the same document summarised for several users, a test suite re-running its prompts. Each repeat costs tokens and seconds. A cache turns a repeat into a lookup that costs nothing and takes under a millisecond, and the provider is never called.
- 2
The cache key. The key must contain everything that can change the answer: the model name, the full prompt (messages, system prompt), the temperature and
max_tokens, and anything else you send. Normalise only what truly doesn't matter, such as extra spaces. A key that leaves something out returns a wrong answer; a key that includes noise (a timestamp, a request id) never hits. The example builds the key from model, temperature, max tokens and the whitespace-normalised prompt. - 3
TTL: how long is it still true. A time to live (TTL) is how long a saved answer stays valid. After it, the entry expires and the next request calls the model again. Pick TTL by how fast the truth changes: minutes for news and prices, days for stable explanations. Clearing a cache entirely when you change the prompt or model is also needed (include a prompt version in the key).
- 4
When caching is wrong. First, temperature above 0: the model is meant to vary; caching makes every user see the same "random" answer. Second, personal data: an answer built from one user's account must never be returned to another. If you do cache personal answers, the user id must be part of the key. Third, answers that depend on the current time or live data ("what is the weather") unless the TTL is shorter than the change. The example shows the cache skipping the temperature 0.8 and personal requests.
- 5
What to measure. The hit rate (hits divided by requests) tells you how much you save. A low hit rate means your requests are too varied for exact matching, which is where semantic caching (Topic 6.5) and provider prompt caching (Topic 6.6) come in. Also log hits so a wrong cached answer can be traced and the entry removed.
- 6
Where the cache lives. Inside one process, a dictionary like the example is simple but lost on restart and not shared between servers. A shared store (such as Redis) with expiry built in is the usual production choice. Set a size limit and an eviction rule (drop the least recently used) so memory doesn't grow without end.
Explain it without notes
What must go into the cache key for a model call, and why?
When should you not cache a model answer? Give three cases.
What does TTL do, and how do you choose it?
What is a cache stampede and how do you reduce it?
Practice
Write cache_key(model, prompt, temperature) that returns the model, the temperature to one decimal and the prompt with spaces collapsed, joined by |. Print the key for the model "small", the prompt " Hello world " (with extra spaces) and the temperature 0.
Write TtlCache(ttl_ms) with put(key, value, now) and get(key, now) returning None when missing or expired. Show a value found at t=500 and gone at t=1500 with a TTL of 1000.
Trade-offs
- ↔
Exact caching is cheap and exact but only helps when requests repeat word for word.
- ↔
A longer TTL saves more but serves older answers; a shorter TTL is fresher but saves less.
- ↔
A shared cache is shared between servers but adds a network hop and a dependency to run.
Done when you can
Done when you can list what goes in a cache key and what is normalised away.
Done when you can explain TTL and pick one for a given feature.
Done when you can name three kinds of request that must not be cached.
Done when you can build a cache with expiry and count its hits.
Done when you can explain a stampede and one way to reduce it.