Topic 11.5
What RAG Costs
In one line
RAG has a one-off cost (embedding the documents), a small monthly cost (storing the vectors) and a per-question cost (embedding, re-ranking and generation). Working out the cost of one question by hand shows which lever saves the most.
Think of it like this
A food truck owner works out what a burger costs: bun, patty, sauce, a share of the gas, a share of the rent. Only then can she tell whether the extra-large burger earns money, or which ingredient to haggle over. A RAG service needs the same sum for one answer, because "it costs a bit" turns into a large number when multiplied by every question of the month.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Price per million tokens
- How providers quote model and embedding prices: dollars for every million tokens processed.
- Input / output tokens
- Input is what you send (the prompt). Output is what the model writes. Output usually costs more per token.
- Cost per question
- The sum of every stage's cost for one answered question.
- One-off cost
- A cost paid once per document change, such as embedding it.
- Lever
- A setting you can change to move the bill, such as k, a cache or the re-ranker.
- Hit rate
- The share of requests served from a cache.
- Quantisation
- Storing vectors with fewer bits per number to save memory, at a small accuracy cost.
Step by step
01Step 1: list the stages that cost money
Embed documents, store vectors, embed the question, re-rank, generate. Each has a unit (tokens, bytes, searches) and a price. Put each price in a variable named with its unit, such as LLM_IN_PRICE_PER_MTOK.
02Step 2: do the one-off and monthly sums
CHUNKS × TOKENS_PER_CHUNK is 6,000,000 tokens. Divide by a million, multiply by the price: 0.12 dollars. For storage, CHUNKS × DIMENSIONS × 4 bytes, converted to GB, times the monthly price.
03Step 3: compute one question, part by part
Prompt tokens are k × tokens per chunk + overhead. Each part is tokens / 1,000,000 × price. Print the parts next to each other, so the largest one is obvious.
prompt_tokens = k * TOKENS_PER_CHUNK + OVERHEAD_TOKENS
generate_in = prompt_tokens / 1_000_000 * LLM_IN_PRICE_PER_MTOK04Step 4: multiply by traffic
cost per question × questions per month × (1 - hit rate). A cache hit costs nothing in this model; in a real one it costs a lookup, which is tiny.
05Step 5: try each lever alone, then together
Change one variable and recompute. The savings of levers are not simply additive: the combined bill (61.28) saves 106.26, a little less than 30.00 + 50.00 + 50.26 = 130.26, because each lever shrinks what the next one can save.
06Step 6: check the sum against a real bill
After a week of traffic compare your estimate with the provider's usage page. A big difference means a hidden cost: retries, longer prompts than you assumed, or output longer than 200 tokens. Update the variables and keep the script.
Try it yourself
- 1
Change the traffic
Set
QUESTIONS_PER_MONTHto 1,000,000. Predict the new base bill before running. Which lever is biggest now? - 2
Make answers longer
Set
ANSWER_TOKENSto 600. Which part of the question cost grows? Does it change which single lever is biggest? - 3
Raise the hit rate
Try a 60 percent hit rate. At what hit rate does the cache save more than the other two levers combined?
Code & diagrams
All prices are example values in named variables. They are not any provider's prices. The sum is the lesson, not the figures.
Expected output
one-off: embed 6000000 tokens = 0.12 dollars
storage: 30720000 bytes of vectors = 0.0077 dollars per month
per question (k=5): embed question 0.0000004 | re-rank 0.0005000 | generate (input) 0.0008750 | generate (output) 0.0003000 | total 0.0016754
base, 100000 questions a month: 167.54 dollars
send 3 passages, not 5 137.54 dollars, saves 30.00
skip the re-ranker 117.54 dollars, saves 50.00
answer cache, 30% hits 117.28 dollars, saves 50.26
all three together 61.28 dollars, saves 106.26
biggest single lever: answer cache, 30% hitsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Mix up thousands and millions
In the Python tab, in per_question, change prompt_tokens / 1_000_000 * LLM_IN_PRICE_PER_MTOK to prompt_tokens / 1_000 * LLM_IN_PRICE_PER_MTOK and run.
Myth vs fact
Myth
Vector storage is the big cost of RAG.
Fact
For most services the model's generation cost dominates the monthly bill. Storage matters at very large corpus sizes, where memory is the cost.
Myth
Cheaper embeddings mean a cheaper system.
Fact
Embeddings are a small part of the per-question cost. Switching embedding model can cost more through re-embedding the whole corpus (Topic 11.8) than it ever saves.
Myth
Caching cost savings are guaranteed.
Fact
They depend on the hit rate, and hit rates depend on how repetitive your questions are. Measure it in production before counting on it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Provider prompt caching (LLM Engineering Topic 6.6) can reduce the cost of a repeated prompt prefix. It helps when the stable part (instructions) is long, less when retrieved passages differ on every question. Check your provider's rules.
- ▸
Track cost per answered question, not only total spend. A refused question costs almost nothing, so a rising refusal rate lowers total spend while hurting users; a falling cost per answer may mean shorter prompts or missing evidence.
- ▸
Watch tail costs: a few huge prompts (long passages, many hits) or runaway outputs can dominate. Set a maximum prompt size and a maximum output length, and alert on both.
- ▸
Include retries and fallbacks: if 5 percent of calls are retried, generation costs 5 percent more than the formula says (LLM Engineering Topic 6.8).
- ▸
Re-embedding is the hidden multiplier of ingestion: skip chunks whose text hash is unchanged (Topic 4.7 covers content hashes), and the one-off cost applies only to what changed.
Remember this
- 1
Why work it out by hand. Prices change and dashboards lag, but the structure of the sum does not. If you know the formula, you can predict what a change does (send fewer passages, add a cache) before you ship it, and you can explain a bill. The example uses made-up prices stored in named variables. They are not any provider's real prices: put your own in.
- 2
Cost 1: ingestion (once, and again on every change). Embedding costs
tokens × price per million tokens. The example corpus is 20,000 chunks of 300 tokens, so 6,000,000 tokens, which at an example price of 0.02 dollars per million tokens is 0.12 dollars. Embedding is cheap per document. It becomes large when you re-embed everything often (Topic 11.8: changing the model) or index millions of chunks. - 3
Cost 2: storage (monthly). A vector of 384 numbers, each a 4-byte float, is 1,536 bytes. For 20,000 chunks that is 30,720,000 bytes, about 0.03 GB, and at an example storage price of 0.25 dollars per GB-month about 0.0077 dollars a month (0.03072 GB × 0.25). Real systems cost more than the raw vectors: the index structure (Phase 3) adds memory, replicas multiply it, and the original text and metadata sit beside it. For big corpora, memory (RAM) is the real bill, which is why smaller vectors and quantisation exist (Topic 5.7 does the memory maths).
- 4
Cost 3: each question. Four parts, with these example prices: 0.02 dollars per million tokens for embedding, 0.50 per 1,000 searches for re-ranking, 0.50 per million input tokens and 1.50 per million output tokens for generation. Embedding the question (20 tokens, almost nothing: 0.0000004 dollars). Re-ranking (0.50 / 1,000 = 0.0005 per search). Generation input (the prompt: 5 passages × 300 tokens + 250 for instructions = 1,750 tokens, 0.000875 dollars). Generation output (200 tokens at a higher example price: 0.0003). The total is 0.0016754 dollars per question, and generation is 70 percent of it. Output tokens cost more per token than input tokens, but the prompt is far longer.
- 5
Multiply, then look for levers. At 100,000 questions a month the example costs 167.54 dollars. Three levers: send 3 passages instead of 5 (saves 30.00 dollars), skip the re-ranker (saves 50.00), or serve 30 percent of questions from an answer cache (saves 50.26). The cache wins in the example because a hit skips every stage, and the others only shrink one part. All three together cost 61.28 dollars. The biggest lever depends on your numbers, which is why you compute it instead of guessing.
- 6
Cost is also a quality decision. Sending fewer passages saves money and may drop the passage that held the answer (Phase 9 measures that). A cache saves money and risks stale answers (Topic 11.3). Skipping the re-ranker saves money and may lower precision (Phase 6). Never accept a cost cut without the evaluation numbers beside it. LLM Engineering Topic 6.1 has the token arithmetic for a single model call; this topic adds the retrieval parts around it.
Explain it without notes
List the costs of a RAG service and say which are one-off, monthly and per question.
Why is generation usually the largest part of the cost per question?
How do you choose between sending fewer passages, skipping the re-ranker and adding an answer cache?
Why can the savings of several cost levers not simply be added?
Practice
Write generation_cost(prompt_tokens, answer_tokens, in_price, out_price) for prices per million tokens and print the cost of a 1,750-token prompt and 200-token answer at 0.50 and 1.50 dollars per million, to six decimals.
Write monthly_saving(cost_per_question, hit_rate, questions) returning what a cache saves in a month, and print it to two decimals for 0.0016754, 30 percent and 100,000 questions.
Trade-offs
- ↔
Fewer passages per question: a smaller bill and a faster answer, with a risk of dropping the right passage.
- ↔
A re-ranker: better ordering at a per-search cost and extra latency (Topic 11.4).
- ↔
An answer cache: the biggest saving for repetitive traffic, with staleness risk (Topic 11.3).
- ↔
A larger, better model: higher quality and a higher bill on the biggest line item.
Done when you can
Done when you can compute the one-off embedding cost of a corpus from its token count.
Done when you can compute the cost of one question from its parts.
Done when you can multiply by traffic and by a cache hit rate.
Done when you can name the biggest lever for a given set of numbers and explain its quality risk.
Done when you can keep prices in named variables with their units.