Topic 6.5
Semantic Caching
In one line
A semantic cache returns a saved answer when a new question means almost the same as an old one, found by comparing embeddings with a similarity threshold. It saves more calls than exact caching, but a threshold set too low returns the wrong answer to a question that only looks similar.
Think of it like this
A help desk where the clerk remembers answers to common questions. "How do I reset my password?" and "I forgot my password, what now?" are different words for the same problem, so the clerk gives the same answer. But "How do I cancel my order?" and "How do I cancel my subscription?" sound alike and need different answers. A clerk who treats them as the same gives a confident wrong reply.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Embedding
- A list of numbers that stands for the meaning of a text. Similar meanings give similar lists.
- Cosine similarity
- A score from 0 (unrelated) to 1 (same direction) comparing two embeddings.
- Threshold
- The lowest similarity score at which two questions count as the same.
- False hit
- A cache hit that returns the answer to a different question.
- Paraphrase
- The same meaning in different words.
- Stop words
- Common words (the, is, how) removed because they carry little meaning.
Step by step
01Step 1: exact caching misses paraphrases
"Can I reset the password?" and "how do I reset my password" have different keys, so an exact cache treats them as new. Support teams see this constantly: the same few intents, endlessly reworded.
02Step 2: turn a question into a vector
Real embeddings come from an embedding model. The example uses a toy: the set of meaningful words left after removing stop words. It is enough to show the mechanics, and nothing in the cache logic depends on how the vector was made.
def embed(text):
return {w for w in text.lower().replace("?", "").split() if w not in STOP}03Step 3: compare with cosine similarity
For two word sets the score is shared words divided by the square root of the two sizes multiplied. Identical sets score 1.0, no shared words score 0. The cache computes the score against every saved entry and takes the best.
04Step 4: run at a strict threshold
At 0.90 the paraphrase "Can I reset the password?" hits (score 1.000) and everything else misses. No wrong answers, but also no savings on "How to change my password", which a person would say is close.
05Step 5: relax the threshold and look for the damage
At 0.60 the same questions give one more hit, and that hit is wrong: "cancel subscription today" matches "cancel order today" with 0.667 because two of three words are shared. Same code, different threshold, a different product.
06Step 6: decide with data, not feelings
Collect real pairs labelled "same" or "different", compute scores, and choose a threshold with no false hits on the labelled set. Then keep logging scores in production. Threshold choice is an evaluation task, not a guess (Phase 9).
Try it yourself
- 1
Predict before you run
Work out the score of "How to change my password" against "reset password" (one shared word out of two and two). Then guess what happens at threshold 0.5.
- 2
Find the breaking threshold
Try thresholds 0.7 and 0.65 in the loop. At what value does the wrong hit first appear? Why exactly at that point?
- 3
Add a safety check
Add a safety rule with a short list of key words (
order,subscription). Refuse a hit when the new question and the cached question contain different words from that list. Does the wrong hit disappear at 0.6, and does the right hit still work?
Code & diagrams
A toy embedder (sets of words) stands in for a real embedding model. The threshold logic is the real thing.
Expected output
threshold 0.90:
'Can I reset the password?' -> HIT 'reset password' (1.000) right
'How to change my password' -> miss (best 0.500)
'cancel subscription today' -> miss (best 0.667)
'Is the office open today?' -> miss (best 0.333)
hits 1, wrong hits 0
threshold 0.60:
'Can I reset the password?' -> HIT 'reset password' (1.000) right
'How to change my password' -> miss (best 0.500)
'cancel subscription today' -> HIT 'cancel order today' (0.667) WRONG
'Is the office open today?' -> miss (best 0.333)
hits 2, wrong hits 1Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Set the threshold too low
Change the thresholds in the loop from (0.9, 0.6) to (0.3,) (in each language use an array with 0.3 only). Run it.
Myth vs fact
Myth
Semantic caching is just a better exact cache.
Fact
Exact caching never gives the wrong answer for a different question. Semantic caching can, so it needs thresholds, tests and monitoring.
Myth
A similarity of 0.9 means the answer is right.
Fact
Scores depend on the embedding model. 0.9 from one model may mean something very different from 0.9 from another. Tune per model.
Myth
Hits are free.
Fact
Each lookup embeds the question and searches the vectors, which costs time and a small amount of money.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Store the original question, the score and the matched question with every hit, and sample them for review. This is how you find your first false hit before a customer does.
- ▸
Scope the cache: partition by product, language and tenant so a question about one product can't match another's. Many false hits are scope errors, not threshold errors.
- ▸
Reranking: use the cache only as a suggestion, and have a small, cheap model confirm "does this answer the new question?" before returning it. This costs a call but removes most false hits.
- ▸
Re-embed everything when you change embedding model. Vectors from different models can't be compared.
- ▸
Estimate the break-even before building it: expected saving = hit rate × cost of a model call, minus the cost of one embedding and search on every request (hits and misses). An embedding call is much cheaper than a large-model call, so a modest hit rate can pay, but a wrong answer costs far more than any saving, so the false-hit rate on labelled pairs decides, not the hit rate.
Remember this
- 1
Why this exists. Exact caching (Topic 6.4) misses paraphrases: people ask the same thing in thousands of different ways, so the hit rate stays low. A semantic cache looks for a saved question that means the same, and reuses its answer.
- 2
How it works. An embedding is a list of numbers that represents the meaning of a text; texts with similar meaning get lists that point in similar directions (Topic 8.3 builds this by hand). The cache stores the embedding of each saved question. For a new question it embeds the text, compares it with the saved ones using cosine similarity (a score from 0 for unrelated to 1 for the same direction), and if the best score is at least a threshold, it returns that entry's answer.
- 3
The threshold is the whole game. Set it high (0.9) and only near-identical questions hit: few false answers, few savings. Set it low (0.6) and more questions hit, including ones that merely share words. The example uses a toy embedder (sets of meaningful words) and shows both settings on the same questions: at 0.60, "cancel subscription today" wrongly reuses the answer for "cancel order today".
- 4
False hits are the risk. An exact-cache miss costs a model call. A semantic false hit gives a user a wrong answer that looks right, and nothing crashes or alerts you. The danger is greatest where small word changes change the truth: numbers, negations ("can" and "can't"), names, dates, products. For answers that must be correct (money, health, legal, security), don't use semantic caching, or require a human-approved question list.
- 5
Make it safer. Cache only questions with stable, generic answers (FAQs). Keep the threshold high and tune it on real labelled pairs: collect pairs that should and should not match, and pick the threshold that gives no false hits on them (Phase 9 covers evaluation sets). Add exact checks on key details (same product id, same number). Log every semantic hit, with both questions and the score, so wrong hits can be found and removed.
- 6
Cost of the cache itself. Every lookup embeds the question (a small model call) and searches the saved embeddings (a vector search, Phase 8). That is much cheaper and faster than a large model call, but not free. If your hit rate is below a few percent, the cache costs more than it saves.
Explain it without notes
What problem does a semantic cache solve that an exact cache doesn't?
What is the role of the threshold, and what happens at the two extremes?
Why is a false semantic hit worse than a miss?
How would you choose a threshold for a real product?
Practice
Write similarity(a, b) for two sets of words returning shared / sqrt(len(a) * len(b)), 0.0 for an empty set. Print it to 3 decimals for {"reset", "password"} and {"change", "password"}.
Write lookup(question_words, cache, threshold) where cache is a list of (word_set, answer) and the function returns the answer of the best entry at or above the threshold, else None. Use the similarity function. Show a hit and a miss.
Trade-offs
- ↔
Semantic caching raises hit rate but adds the risk of wrong answers and the cost of embedding and searching.
- ↔
A high threshold is safe and saves little. A low threshold saves much and misleads users.
- ↔
A confirming check with a small model removes most false hits but adds back some latency and cost.
Done when you can
Done when you can explain how a semantic cache finds a hit.
Done when you can show a false hit and the threshold that caused it.
Done when you can name the kinds of question that must not use semantic caching.
Done when you can describe how to choose and monitor a threshold.