Topic 13.4
Caching AI Responses
In one line
A cache can answer the same general question for free. It must key on a normalised question and the facts it depends on, expire or be replaced when facts change, replay in the same streaming format, and never hold personal or incomplete answers.
Think of it like this
A tea stall with a handwritten board: "Opening hours 7 to 10". The owner answers that question from the board instead of thinking about it each time. But the board must be rubbed out and rewritten when the hours change, and the owner never writes "your parcel arrived" on it, because that answer is for one customer only. Cache the board answers. Never cache the private ones.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Cache
- A store of saved answers returned instead of doing the work again.
- Cache key
- The label an answer is saved under. The same key finds the same answer.
- Normalise
- Make equal things look equal: trim spaces, lowercase, remove trailing punctuation.
- Hit / miss
- A hit is found in the cache. A miss is not found, so the model is called.
- Stale
- Found, but older than allowed. It must be refreshed.
- TTL (time to live)
- How long a saved answer may be used before it counts as stale.
- Invalidation
- Removing or ignoring saved answers when the facts behind them change.
- Prompt caching
- A provider feature that charges less for input it has recently seen. It does not skip the model call.
Step by step
01Look up before you ask
The route builds a key, looks it up, and decides: no entry is a miss, an entry older than the TTL is stale, anything else is a hit. A hit streams the saved text. A miss or stale calls the model and, when the stream ends well, saves the new answer.
02Hit, miss, stale, changed facts
The first example asks the same question four times. The server logs the key and the state. Ask 1 is a miss (model call 1). Ask 2 spells the question differently, normalises to the same key and is a hit: the answer still says "model call 1", proving the model was not called. At 11 minutes the entry is stale, so the model is called again (model call 2). After shopInfoVersion becomes 2 the key changes, so it is a miss again (model call 3).
03What must never be cached
The second example runs two routes. Version A caches everything under the question alone: Asha asks "Where is my order?" and Ravi's later "where is my order" is a hit that shows Asha's order. Version B skips personal questions (skipped: personal), caches the general return-window answer (miss then hit), and refuses to save an essay that ended with finish reason length (miss, not saved: length, then a miss again for the next user).
04A cached answer must still stream
On a hit the first example writes the saved text word by word with createUIMessageStream. The page's code is identical for hits and misses. The only difference is the x-cache header, which the page shows as a small label and your logs record.
Try it yourself
- 1
Shorten the TTL
In the first example set
TTLto0and changeminute = 11;tominute = 1;. The second ask is still a hit at minute 0, but the third becomes stale after one minute. A short TTL keeps facts fresh and saves less. - 2
Add the language to the key
In the first example add
let language = "en";abovekeyFor, changekeyForto(q) => "v" + shopInfoVersion + ":" + language + ":" + normalise(q), and replaceshopInfoVersion = 2;withlanguage = "hi";. The last ask uses the keyv1:hi:how long is shippingand is a miss (model call 3). Answers in different languages must never share a cache entry. - 3
Make every question personal
In the second example change the
isPersonalpattern to/./. Every question skips the cache and the hit lines disappear. That is the cost of being safe by default, and also the right place to start when you are unsure.
Code & diagrams
The "(model call N)" text comes from the stand-in model's call counter. When it does not change, the model was not called. In a real app the Map is a shared store such as Redis.
Preview
Console
Nothing logged yet.Expected console output
(server) key "v1:how long is shipping" → miss
first ask → cache: miss | Shipping takes 1 to 2 working days. (model call 1)
(server) key "v1:how long is shipping" → hit
same question, other spelling → cache: hit | Shipping takes 1 to 2 working days. (model call 1)
(server) key "v1:how long is shipping" → stale
11 minutes later → cache: stale | Shipping takes 1 to 2 working days. (model call 2)
(server) key "v2:how long is shipping" → miss
after the shop info changed → cache: miss | Shipping takes 1 to 2 working days. (model call 3)Line 2 is the leak: Ravi is shown Asha's order. The personal-question test here (a regular expression) is the simplest possible. A real app decides from how the answer was produced: did the route read any user data?
Preview
Console
Nothing logged yet.Expected console output
a asha "Where is my order?" → [miss] Your order ORD-1042 (Masala chai 250 g) has shipped.
a ravi "where is my order" → [hit] Your order ORD-1042 (Masala chai 250 g) has shipped.
---
b asha "Where is my order?" → [skipped: personal] Your order ORD-1042 (Masala chai 250 g) has shipped.
b ravi "where is my order" → [skipped: personal] Your order ORD-1043 (Ginger biscuits) is being packed.
b asha "What is the return window?" → [miss] Returns are accepted within 7 days.
b ravi "what is the return window" → [hit] Returns are accepted within 7 days.
b asha "Write an essay about tea" → [miss, not saved: length] Tea began in China and then
b ravi "Write an essay about tea" → [miss, not saved: length] Tea began in China and thenBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the version in the key
In the first example change keyFor to const keyFor = (q) => normalise(q);.
Break #2
Cache by question alone
Use version A from the second example for a signed-in feature: it already caches every answer under the question alone.
Myth vs fact
Myth
Caching AI answers is always a good idea.
Fact
It is a good idea for general, stable answers. For personal, live or one-off answers it is a bug waiting to happen.
Myth
Prompt caching and response caching are the same.
Fact
Prompt caching lowers the price of repeated input but the model still runs. A response cache skips the model call entirely.
Myth
The same question always deserves the same answer.
Fact
Models vary on purpose. Serving a saved answer makes your product more consistent, and also hides any later improvement until the entry expires.
Myth
A hit should be instant.
Fact
A hit should stream like a live answer, so the page behaves identically. Dumping a saved answer all at once changes the feel of the screen.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
A cache stampede happens when a popular entry expires and a hundred users ask at once: all miss, all call the model. Fix it by letting one request refresh while others get the old answer ("stale while revalidate"), or by locking per key.
- ▸
Log the cache state (
hit,miss,stale,skipped) with every request, never the question. The hit rate tells you if the cache is worth keeping, and a sudden drop means a key bug. - ▸
Semantic caching (matching questions that mean the same, not just spell the same) saves more and is risky: two different questions can look close. Start with exact normalised keys. If you try semantic matching, test with near-miss pairs such as "return an opened item" and "return an unopened item".
- ▸
Decide personal versus general before the model runs, from how the request is built (did the route read the user's data or call a user-specific tool?), not from the user's wording. Wording tests are easy to fool and easy to get wrong.
Remember this
- 1
The problem this solves: Many users ask the same general question ("what is your return policy?"). Paying a model, and waiting for it, every time is wasteful. A cache returns the saved answer in a few milliseconds for no cost. But a careless cache serves stale facts, or worse, one user's private answer to another. This topic shows what is safe, how to key it, and how to keep it fresh.
- 2
What is safe to cache. A general answer that depends only on the question and on facts you control (shop policy, shipping times, product descriptions). Never cache an answer that used personal data (an order status, an address, anything about the signed-in user), an answer that used a tool with live data, an incomplete answer (finish reason other than
stop), or an error. The second live example shows all four rules. The leak in version A is the most expensive bug in this topic. - 3
The key. A cache is a map from a key to a saved answer. If the key is the exact text, "How long is shipping?" and "how long is shipping " are two entries. Normalise the question first: trim, lowercase, collapse spaces, drop the trailing "?" or ".". Then add everything the answer depends on to the key: a version number of your shop facts, the language, and the model or prompt version, so a changed prompt does not serve answers written by the old one.
- 4
Freshness. Saved answers go out of date. Two tools: a time to live (TTL), after which an entry is treated as stale and refreshed, and a version in the key that you bump when facts change, so old entries are never matched again. The first example shows both: an entry 11 minutes old (TTL 10) is "stale" and the model is asked again, and a changed
shopInfoVersionmakes the same question a miss. - 5
Streaming a cached answer. Users and the page expect the same stream format as a live answer. On a hit, the route builds a UI message stream with
createUIMessageStreamand writestext-start, onetext-deltaper piece, thentext-end. The page cannot tell the difference, and the text still appears piece by piece. Sending it all at once would make cached answers feel different from live ones, and it can break layouts that expect streaming. A header such asx-cache: hit | miss | staletells the page and your logs what happened. - 6
Save only when the stream ended well. Save in the model call's
onEndcallback, and only whenfinishReason === "stop". A stream that was cut by the length cap, broken by an error or stopped by the user would otherwise be cached, and the next hundred users would read a half answer. That check does not look at which model answered. If you want only full-quality answers saved, also test the mode, as the build topic does (mode === "main"). - 7
Follow-up questions are not general. The first example builds its key from the last message only, which is right for a first question such as "How long is shipping?". A follow-up such as "And for returns?" or "What about the second one?" means something only because of the earlier turns. Cached under its own words, it would give one user's answer to a different user's different conversation. Cache only the first question of a conversation (a one-message history), or put a hash of the whole history into the key, which makes hits rare.
- 8
Where the cache lives. A JavaScript
Mapin the route works in one process only. On a serverless host each request may run in a new process, so a real cache is Redis, a database, or the platform's cache. Providers also offer prompt caching (cheaper repeated input), which is separate: it reduces what you pay for the model, while this topic's cache avoids calling the model at all. See/system-designfor cache invalidation and stampedes.
Explain it without notes
What is safe to cache in an AI feature, and what must never be cached?
How do you build a good cache key?
How does the page keep working the same way when an answer comes from the cache?
How do you keep cached answers from going stale?
Practice
Add a counter to the first example's server that logs the hit rate (hits divided by all requests) after each ask, using whole percents.
In the second example's version B, make the cache private per user for questions that mention "my order", instead of skipping the cache. What must the key contain, and why is this riskier than skipping?
Trade-offs
- ↔
A cache cuts cost and latency and adds a second source of truth that can be wrong. Short TTLs and versions reduce the risk and reduce the savings.
- ↔
Caching only general answers keeps the feature simple and safe, and leaves the most expensive personal answers uncached. That is usually the right first version.
Done when you can
I can say what is safe to cache and name four things that never are.
I can build a normalised key that includes a facts version.
I can explain hit, miss and stale, and when each happens.
I can stream a cached answer in the same format as a live one.
I can save an answer only when the stream ended with a normal stop.