Topic 11.3
Caching Every Stage
In one line
Most questions are repeats, so each stage of the pipeline (embedding, retrieval, the final answer) can remember its result. The hard part is the key: it must contain everything that changes the result, so a changed document, model or prompt never serves an old answer.
Think of it like this
A librarian is asked the same question twenty times a day: "Where are the cookery books?" After the third time she writes the answer on a card and tapes it to the desk. That is a cache. It works until the cookery section moves. If nobody removes the card, the next twenty visitors walk to the wrong shelf with total confidence.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Cache
- A fast place that remembers the result of slow work so repeats can skip it.
- Cache key
- The label an entry is stored under. Two requests share an entry only if their keys are equal.
- Hit / miss
- A hit finds the stored result. A miss finds nothing and must do the work.
- Invalidation
- Removing or ignoring entries that are no longer correct.
- TTL (time to live)
- How long an entry may be used before it is dropped, whatever else happens.
- Normalisation
- Turning different spellings of the same question into one standard form before keying.
- Dependency map
- A record of which cached entries were built from which documents.
- Semantic cache
- A cache that matches by similarity of meaning instead of exact key. Fast to try, risky to trust.
Step by step
01Step 1: list what each stage reads
Embedding reads the text and the model. Retrieval reads the vector, the index contents, k and filters. The answer reads the passages, the prompt template and the answering model. Each item on the list must appear in that stage's key.
02Step 2: build normalise() and key_for()
Lower-case, keep only letters and digits, join with single spaces. Then join the parts with |. The example's answer key looks like toy-v1|index7|p3|what does the monthly pass cost.
def key_for(stage, question):
q = normalize(question)
return STATE["embed_model"] + "|index" + str(STATE["index_version"]) + "|" + STATE["prompt_version"] + "|" + q03Step 3: look up top-down
Try the answer cache first. On a miss try the retrieval cache, then the embedding cache. A hit at a higher stage skips everything below it. The example prints the stages it tried, so a first ask shows three misses and a repeat shows one hit.
04Step 4: change one ingredient at a time
A new prompt version misses the answer cache but hits retrieval. A new index version misses answer and retrieval but hits embedding (the question's vector has not changed). A new embedding model misses everything. That ladder is exactly what the keys are designed to produce.
05Step 5: handle an edited document
Version keys handle it for free but throw away everything. For precise invalidation record deps[doc_id] = {question keys} when you store an answer, and on an edit drop those keys. The second example edits D8 and drops only the price answer, keeping the opening-hours answer.
06Step 6: add a TTL and a size limit
Every entry expires after a period you choose (an hour, a day), and the cache has a maximum size so it cannot eat memory. When it is full, drop the least recently used entries. Both are settings of the cache you use (a Redis server, an in-process LRU), not code you write by hand.
Try it yourself
- 1
Predict step 4
Before running, write down for each of the five asks which of answer, retrieve and embed will hit. Then run and compare. Which row surprised you?
- 2
Add the filter
Add a
tenantfield toSTATEand put it into the retrieve and answer keys. Why would two customers asking the same question need different entries? - 3
Invalidate too much
In the second example, make
editclear the wholeCACHE. Does the opening-hours answer still survive? What did the precise version save?
Code & diagrams
The caches only remember that a key was seen. The expensive work is simulated by the WORK counters.
Expected output
answer key: toy-v1|index7|p3|what does the monthly pass cost
1 first ask: answer=miss retrieve=miss embed=miss
2 same, other case: answer=hit
3 prompt changed: answer=miss retrieve=hit
4 index changed: answer=miss retrieve=miss embed=hit
5 model changed: answer=miss retrieve=miss embed=miss
real work done: embed=2 retrieve=3 answer=4Each answer remembers which document it used. On an edit the entries that used it are dropped and nothing else is.
Expected output
invalidate on edit: no
The monthly pass costs 25 dollars. (fresh)
Shops open at 7 am. (fresh)
The monthly pass costs 25 dollars. (cached)
Shops open at 7 am. (cached)
invalidate on edit: yes
The monthly pass costs 25 dollars. (fresh)
Shops open at 7 am. (fresh)
dropped cache entry: pass price?
The monthly pass costs 30 dollars. (fresh)
Shops open at 7 am. (cached)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Leave the index version out of the retrieval key
In the Python tab, change the retrieve key line to return STATE["embed_model"] + "|k3|" + q (no index version) and run.
Myth vs fact
Myth
Caching an LLM answer is always safe.
Fact
It is safe only when the key contains the question, the sources' versions, the prompt and the model, and the user's permission scope. Miss one and you serve a wrong or even a private answer.
Myth
A semantic cache (match similar questions) is just a better cache.
Fact
Similar is not same: "cancel 2 hours before" and "cancel 2 hours after" are close in meaning space and have opposite answers. Use exact keys first; if you add similarity matching, use a high threshold and measure wrong hits.
Myth
Caches only matter for cost.
Fact
They are also a latency tool and a protection: a hit skips a slow, rate-limited provider call entirely.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Add single-flight (request coalescing): if fifty identical questions arrive while the first is being computed, let one do the work and share its result. Without it, a cache miss on a popular question becomes a stampede on the model.
- ▸
Track hit rate per stage as a metric (Topic 11.6). A sudden drop in the embedding hit rate usually means a normalisation change or a key bug; a drop in the answer hit rate right after a deploy usually means a version bump, which is expected.
- ▸
Embeddings are a good cache target for documents, too: key by
(model, chunk text hash)and re-embedding after a small edit only costs the chunks whose text changed. Ingestion cost (Topic 11.5) drops sharply for documents with small edits. - ▸
Negative results need care: cache a refusal briefly, and never cache a timeout or an error as if it were an answer.
- ▸
LLM Engineering Topics 6.4 and 6.5 cover exact and semantic caching of model calls, and System Design covers eviction (LRU, TTL), stampedes and cache-aside in general. This topic is about the keys of a whole retrieval pipeline and how a document change reaches them.
Remember this
- 1
Why cache at all. Repeats are common, and each stage is expensive in its own way: embedding costs a model call, retrieval costs index time, generation costs the most money and the most seconds. A hit skips the work entirely. A good cache cuts both the bill (Topic 11.5) and the wait (Topic 11.4).
- 2
One cache per stage, each with its own key. The embedding cache maps
(model, normalised text)to a vector. The retrieval cache maps(model, index version, k, filters, normalised query)to a list of passage ids. The answer cache maps all of that plus the prompt version to the final text. A higher stage hit skips every stage below it. Stages differ in how often they hit: the embedding cache hits whenever a query repeats; the answer cache hits only when everything else is also unchanged. - 3
A key must contain everything that changes the value. If the answer depends on the index contents, the key contains the index version. If it depends on the prompt template, the key contains the prompt version. If it depends on the embedding model, the key contains the model name. A key missing one of these is a bug that returns wrong answers silently. The example changes each ingredient in turn and shows which cache stops matching.
- 4
Normalise before keying. "What does the monthly pass cost?" and "what does the MONTHLY pass cost" should be one entry. Lower-case, collapse spaces, strip punctuation. Keep normalisation light: if you also drop words or stem, two different questions may share an entry. The goal is to merge only the questions that are truly the same.
- 5
Invalidation: the part that is hard. When a document changes, every cached answer built from it may be wrong. There are two designs. Version keys: put the index version in every key, so a new version simply never matches old entries (they age out). Simple and safe, but one edit invalidates everything. Dependency tracking: record which documents each entry used, and on an edit delete exactly those entries. Precise, but you maintain a reverse map. The second example shows a stale answer without it and a correct one with it.
- 6
Also set a time limit. Even with precise invalidation, give entries a time-to-live (TTL), so a missed event cannot leave a wrong answer forever. And cache only what is safe: never cache across users when answers depend on permissions, unless the user's access scope is in the key (Topic 10.4). Do not cache failed or refused answers for long, or one outage becomes a long one.
Explain it without notes
What should the cache key of a RAG answer contain, and why?
Compare version keys with dependency tracking for invalidation.
Why is a semantic cache risky?
Which stages of a RAG pipeline are worth caching, and in what order do you look them up?
Practice
Write cache_key(stage, model, index_version, question) that normalises the question (lower-case words joined by single spaces) and returns "embed|<model>|<q>" for stage embed and "<stage>|<model>|index<version>|<q>" otherwise. Show that two spellings of one question share a key and that a new index version changes the retrieval key but not the embedding key.
Write invalidate(cache, deps, doc_id) that removes from cache every key listed in deps[doc_id], removes the deps entry, and returns how many keys were removed (0 when the document is unknown).
Trade-offs
- ↔
Aggressive caching: lower cost and latency, more ways to serve a stale or wrong answer.
- ↔
Version keys: trivially correct and a heavy hammer; dependency tracking is precise and needs upkeep.
- ↔
Long TTLs: higher hit rate, longer exposure to a missed invalidation.
Done when you can
Done when you can name the three caches of a RAG pipeline and what each key contains.
Done when you can predict which caches miss after a change of prompt, index or model.
Done when you can explain two invalidation designs and when to use each.
Done when you can say why a semantic cache needs extra care.
Done when you can list things that must be in a key when users have different permissions.