Topic 6.6
Provider Prompt Caching
In one line
Many providers remember the work they did on the start of your prompt and reuse it when a later request begins with the same text, which makes repeated long prompts cheaper and faster to start. You benefit by putting the stable text first and the changing text last.
Think of it like this
A student who read the first 30 pages of a booklet yesterday, and you hand the booklet over again today. If only the last page is new, the student reads just that page. If a new page is slipped in at the front, the student can no longer trust anything they remember, and must read it all again from the start. A prompt cache works the same way: it remembers reading the start of your prompt, and the memory only helps while the start is unchanged.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Prefix
- The start of a prompt, counting from its first token.
- Prefix match
- The longest start that two prompts share exactly. Only this part can be reused.
- Prompt caching
- A provider feature that reuses the work done on a repeated prefix.
- Cached tokens
- Input tokens served from the prompt cache, reported in usage and billed differently.
- Minimum prefix size
- The shortest prefix a provider will cache. Shorter ones are never reused. It differs by provider.
- Cache expiry
- Prompt caches forget a prefix after a period without use.
Step by step
01Step 1: spot what repeats
Look at your prompts. Which text is the same in nearly every request? Rules, a persona, a document, examples, tool definitions. That repeated start is what a prompt cache can reuse.
02Step 2: build a fake provider
The fake remembers the last prompt as a list of tokens. For a new prompt it counts how many tokens match from the start, and only counts them as cached if the match is at least the minimum size and the entry has not expired. This is a simplification (real systems keep many prefixes) but it shows the rule that matters: matching starts at token zero.
shared = 0
while shared < min(len(tokens), len(self.prefix)) and tokens[shared] == self.prefix[shared]:
shared += 1
cached = shared if shared >= MIN_CACHEABLE else 003Step 3: stable text first
The first request reads all 305 tokens fresh. The next two share the 300-token policy and differ only in the five question tokens, so 300 tokens are reused. The cost drops by almost half in the example prices because reused tokens are priced lower.
04Step 4: question first
Now put the five question tokens at the start. The second request's first token already differs from the first request's, so the match is zero and nothing is reused, although 300 identical tokens follow. Same text, same provider, no saving.
05Step 5: watch expiry
The last request comes 380 seconds after the previous one in simulated time. The example's cache lifetime is 300 seconds (an example number), so the entry is gone and the whole prompt is read fresh again. Prompt caching rewards steady traffic.
06Step 6: measure, don't assume
Read the cached-token field from each real reply and record the share of input tokens that were cached. Compare it with what your ordering should give. The Raw HTTP view in this course's labs shows the real usage block of a reply; look for fields beyond the plain token counts.
Try it yourself
- 1
Predict the saving
For "stable text first", compute the cost of request 2 by hand: 5 fresh tokens at price 2 plus 300 cached at price 1. Run to check.
- 2
Raise the minimum
Set
MIN_CACHEABLEto 400. What happens to the cached counts? What does this teach about short system prompts? - 3
Add a timestamp
Add a token
now=<time>at the very start ofstable_first(for example[f"now={now}"] + POLICY + ...; passnowin). What happens to the cached counts, and where should the timestamp go instead?
Code & diagrams
A fake provider that remembers the last prompt. Prices, the minimum size and the expiry time are example numbers: real providers differ and change them.
Expected output
stable text first:
t=0: 305 tokens, cached 0, cost 610 micro-dollars
t=10000: 305 tokens, cached 300, cost 310 micro-dollars
t=20000: 305 tokens, cached 300, cost 310 micro-dollars
t=400000: 305 tokens, cached 0, cost 610 micro-dollars
total 1840 micro-dollars
question first:
t=0: 305 tokens, cached 0, cost 610 micro-dollars
t=10000: 305 tokens, cached 0, cost 610 micro-dollars
t=20000: 305 tokens, cached 0, cost 610 micro-dollars
t=400000: 305 tokens, cached 0, cost 610 micro-dollars
total 2440 micro-dollarsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Put a changing value at the start
Give both builder functions a second argument now, call build(question, now), and make stable_first start with an id token: return [f"id{now}"] + POLICY + question.split() (do the same in every language). Run it.
Myth vs fact
Myth
Prompt caching stores the answer.
Fact
It stores work done on the input. The model still writes a new answer every time.
Myth
The same text anywhere in the prompt gets cached.
Fact
Only a shared start counts. One different token early in the prompt ends the match.
Myth
Every provider caches automatically on the same terms.
Fact
Some cache without being asked and some need you to mark the part. Pricing, minimum sizes and lifetimes differ. Check the current documentation.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Order the prompt by how often each part changes: tool definitions and system prompt, then fixed reference text, then rarely changing examples, then history, then the new message. This also matches how a conversation grows by appending.
- ▸
Appending to history keeps the earlier prefix intact, which is good. Editing or summarising earlier messages in place changes the prefix and loses the benefit. Plan trimming (Topic 3.7) with this in mind.
- ▸
Keep tool lists and settings stable between calls: a reordered tool list is a different prefix.
- ▸
Warm-up and pings can keep a prefix alive, but you pay for them. Do the arithmetic against what a cold start costs.
Remember this
- 1
Why this exists. Many requests start with the same long text: a system prompt with rules, a product manual, a few examples, the same document asked about ten times. Before the model can write, it reads (prefill, Topic 6.3) all of that input again. A provider can save the result of reading a text and reuse it, so the repeated part costs less and starts faster.
- 2
The mechanism: a stable prefix. The provider matches the start of your new prompt against recently used prompts. The longest part that is identical from the first token is a prefix match and can be reused. The first token that differs ends the match, and everything after it is read fresh. This is why order matters: the example shows the same 300 tokens of fixed text and a five-token question, once with the fixed text first (almost all reused) and once with the question first (nothing reused).
- 3
What the provider charges. Reused tokens are usually billed differently from fresh ones, and the reply's
usagereports how many were read from the cache. How they bill, the minimum prefix size, how long an entry lives, and whether you must mark the cacheable part yourself all differ by provider and change over time. This course states no numbers for them. Read your provider's current documentation, and in your code keep those values in configuration like any price. - 4
Put stable text first, variable text last. System prompt and rules first, then fixed reference text and examples, then conversation history, then the new user message last. Don't put a timestamp, a user name or a request id at the top: one changing character at the start kills the match for everything after it. If a changing detail is needed, put it near the end.
- 5
It is not your cache. A prompt cache does not skip the model: the model still writes a fresh answer. It reduces cost and delay of reading the input. So it differs from exact caching (Topic 6.4), which skips the call, and from semantic caching (Topic 6.5), which reuses an answer. It also forgets: entries expire after a short time without use, so a feature used rarely gets less benefit than one used constantly (the last request in the example comes after expiry and gets nothing).
- 6
How to verify it works. Log the cached-token count the provider reports for each call. If your hit share is lower than expected, look for text that changes near the start of the prompt, a prefix shorter than the provider's minimum, calls too far apart, or a different model name or tools list (anything that changes before the long text). Measure before assuming savings.
Explain it without notes
What does a prompt cache save, and how does it differ from an exact-match cache?
Why does the order of parts in a prompt matter for prompt caching?
How would you check that prompt caching is actually working in your product?
Practice
Write common_prefix_length(a, b) for two lists and show it for [1, 2, 3, 4] with [1, 2, 9, 4], and for [5, 6] with [1, 6].
Write cost(total_tokens, cached_tokens, price_fresh, price_cached) returning the cost in micro-dollars (prices per million tokens, so tokens times price). Print it for 305 total, 300 cached, prices 2 and 1.
Trade-offs
- ↔
Prompt caching cuts cost and delay for repeated prefixes but constrains how you can order and edit prompts.
- ↔
Keeping a prefix warm costs calls; letting it expire costs a cold start. Steady traffic gets the best result.
- ↔
Provider-specific features tie you to one provider's rules; keep the layout provider-neutral and the settings in configuration.
Done when you can
Done when you can explain prefix matching and why order matters.
Done when you can reorder a prompt so the stable text comes first.
Done when you can say how prompt caching differs from exact and semantic caching.
Done when you know where your provider reports cached tokens and have logged it.