Topic 12.8
Long Context Windows vs RAG
In one line
When the whole corpus fits in the model's context window, you can paste it instead of searching. That is simpler, but costs input tokens on every question. Work out the numbers, and consider using both.
Think of it like this
If you have a one-page recipe, you do not need an index. You put it on the counter and read it. For a shelf of 500 cookbooks you need the index. Pasting is the one-page case and RAG is the shelf.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Context window
- The most text, counted in tokens, a model can read in one request.
- Input token
- A token you send. Billed on every request.
- Break-even
- The point where two options cost the same.
- Prompt caching
- A provider feature that makes a repeated, unchanged beginning of a prompt cheaper. Rules differ by provider.
- First-token latency
- The time from sending a request until the first part of the answer arrives.
- Example price
- A made-up price used for arithmetic. Never use it for budgeting.
Step by step
01Step 1: write the question as numbers
How big is the corpus in tokens? How many questions per month? What does an input token cost? Does it fit the window?
02Step 2: price pasting
Cost of pasting is corpus tokens times price times the number of questions. The same corpus is sent each time.
PRICE_PER_MTOK = 1.00 # example price, check your provider
paste_cost = corpus_tokens * PRICE_PER_MTOK / 1000000 * questions03Step 3: price RAG
Cost of RAG is the retrieved prompt (passages, instructions, question) times price times questions. Add the cost of embedding and storage separately (Topic 11.5).
04Step 4: read the table
The handbook is cheaper to paste. The policy library is cheaper with RAG. The company-wide corpus does not fit at all.
05Step 5: add what the numbers leave out
Latency, the chance of missing a buried fact, permissions per user, citations and freshness. These can overrule the cost table.
06Step 6: decide, then re-check
The right answer changes as the corpus grows and prices change. Keep the calculation in code and re-run it.
Try it yourself
- 1
Change the price
Set
PRICE_PER_MTOKto 0.10 and run. Which verdicts change and which stay? - 2
Change the volume
Set
QUESTIONSto 100000. Does the handbook still favour pasting? Why do the numbers move together? - 3
Add your own corpus
Add a corpus of 40,000 tokens. Predict the verdict first, then run.
Code & diagrams
The price and window size are examples, not any provider's. Change them and watch the verdicts move.
Expected output
handbook 1500 tokens | paste 1.50 | rag 3.00 | paste it
policy library 150000 tokens | paste 150.00 | rag 3.00 | RAG (cheaper)
all company docs 5000000 tokens | paste n/a | rag 3.00 | RAG (does not fit)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the window
In the Python tab, change if corpus_tokens > CONTEXT_LIMIT: to if False:. Run.
Myth vs fact
Myth
Long context makes RAG obsolete.
Fact
It removes RAG for small corpora. Large, changing, permissioned corpora still need retrieval, and cost grows with every question.
Myth
Pasting is always more accurate because nothing is missed.
Fact
Nothing is missed by retrieval, but the model may still overlook a fact in a very long prompt. Test it.
Myth
RAG is always cheaper.
Fact
For a small corpus the RAG prompt can be larger than the corpus, and the system has more parts to run.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Where your provider supports caching a repeated prompt prefix, a pasted corpus can cost much less than the table shows. Read the current rules and test, because they differ and change.
- ▸
Treat a pasted corpus the same as retrieved text for injection and permissions: everything in the prompt is visible to the model for every user who asks.
- ▸
A hybrid is common: retrieve whole documents or large sections (parents, Topic 12.1) rather than tiny chunks, and paste those.
Remember this
- 1
Why this exists again. Context windows (how much text a model can read at once; LLM Engineering Topic 1.5) have grown a lot. For a small corpus RAG may be extra machinery: chunking, embedding, an index, a retriever and a threshold, all things that can fail. If everything fits, why not paste it all? Sometimes you should.
- 2
Cost. You pay for input tokens on every request. Pasting a 1,500-token handbook for 1,000 questions at the example price of 1.00 dollar per million tokens costs 1.50 dollars. RAG with a 3,000-token prompt costs 3.00 dollars: here pasting is cheaper, because the handbook is smaller than the RAG prompt. A 150,000-token library costs 150.00 dollars to paste for 1,000 questions, 50 times the RAG cost. The prices and the window size in the example are variables. Check your provider, and note that cached prefixes can be cheaper on some providers, which moves the line.
- 3
Fit. A corpus over the window simply does not fit, whatever the cost. The 5,000,000-token example is not an option for pasting. Leave room for the question, the instructions and the answer, and do not forget that your corpus grows.
- 4
Latency. A model must read every input token before it answers. Pasting 150,000 tokens is slower than reading 3,000, on every question. Measure first-token latency on your provider with your sizes (Topic 11.4).
- 5
Accuracy. More text is not always better. Models can miss or confuse a fact buried in a very long prompt, and unrelated text can distract. Whether this matters depends on the model and the task, so test with your own questions. Retrieval narrows the text to what is relevant, which can help, but only if it finds the right passages. Be aware of the trade: pasting cannot miss a passage, while RAG can.
- 6
Other differences. Pasting has no index to build or keep fresh, and no per-user filter, so you cannot paste only what a user may see unless you build the prompt per user. RAG can cite passages and filter by permission. For citations, long-context prompts need the same numbered-source layout as RAG (Topic 8.1). Using both: retrieve a generous set (or whole documents) and paste those, so a retrieval miss on a chunk is softened and the prompt stays small.
Explain it without notes
When might pasting the whole corpus beat RAG?
How do you compare their costs?
What non-cost factors can decide the choice?
How can you use both?
Practice
Write cost_to_paste(tokens, price_per_mtok, questions) returning the dollars spent on input tokens when the whole corpus is sent with each question. Print it with 2 decimals for 150000 tokens, a price of 1.00 per million tokens (an example price) and 1000 questions.
RAG costs 0.003 dollars per question and pasting costs 0.150 dollars per question. Building the RAG index costs 20.00 dollars once. Write break_even(paste, rag, index_cost) returning the smallest whole number of questions after which RAG is cheaper overall, and print it. Use math.ceil.
Trade-offs
- ↔
Pasting: simplest, no misses by retrieval, cost and latency grow with the corpus and the questions.
- ↔
RAG: cheap per question at scale, more parts and a retrieval risk.
- ↔
Both: balanced, with extra design.
Done when you can
Done when you can compute the break-even between pasting and RAG for a corpus.
Done when you can say why fit is checked first.
Done when you can name three non-cost factors.
Done when you can describe a hybrid.