Topic 8.2
The Context Budget
In one line
Everything in the prompt competes for one fixed window, so sources get a token budget. You count tokens with a stated rough rule, pack passages best-first, decide what to do with one that does not fit, and order the rest. More context is not better.
Think of it like this
Packing a suitcase for a flight with a weight limit. You cannot take everything. You pack the things you will use most first, and when a big item does not fit you decide whether to stop packing or to look for something smaller that does. A heavy bag is not a better trip, and a bag crammed with the wrong things is worse than a light one.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Token
- The unit a model reads: roughly a short word or a piece of one.
- Context window
- The most tokens a model can read in one request, inputs plus outputs.
- Budget
- A limit you set on purpose, here the tokens allowed for sources.
- Overhead
- Tokens spent on tags and labels around a passage rather than on its text.
- Greedy packing
- Taking items in order and keeping each one that still fits.
- Rank order
- The list from best match to worst, as the retriever returns it.
- Rough rule
- A quick estimate, here 4 characters per token, good for planning but not for the final check.
Step by step
01Step 1: estimate tokens
Write one function that turns a text into a token estimate. Use the provider's tokenizer in production; for planning and tests, the 4-characters rule is fine as long as it is stated.
def estimate_tokens(text):
return (len(text) + 3) // 402Step 2: charge overhead
Each passage also costs its tags. The example uses 8 tokens per passage. Forget it and your budget is a lie (see Break it).
03Step 3: walk down the ranking
Keep each passage whose cost still fits in what is left. When one does not, the policy decides: stop, or skip it and try the next.
04Step 4: compare stop and skip
With a budget of 105, D1 and D3 fit (78 tokens). D12 costs 38 and does not. Stopping ends there. Skipping goes on and keeps the short D9. The example prints both so you can see the difference.
05Step 5: choose the order
The second example arranges the same five passages best-first and best-at-both-ends. A toy reader that skims only the first and last passage sees different passages in the two orders.
06Step 6: tune from data
The budget, the policy and the order are all settings. Change one, run your golden set (Topic 9.2), and keep the change only if the answers improve.
Try it yourself
- 1
Move the budget
Change
BUDGETto 80 and then to 140. For each, predict which passages each policy keeps, then run. At which budget do both policies give the same answer? - 2
Make a passage big
Add a 600-character passage as the first hit. What happens to the other passages under 'stop'? Under 'skip'? Should one huge passage ever be able to push out the rest?
- 3
Reorder
In the second example, add a sixth passage and run. Where does rank 6 land with best-at-both-ends? Write the rule for any number of passages.
Code & diagrams
The 4-characters-per-token rule is a planning estimate. Use your provider's tokenizer for the final check.
Expected output
D1 31 tokens + 8 overhead
D3 31 tokens + 8 overhead
D12 30 tokens + 8 overhead
D9 16 tokens + 8 overhead
stop at the first miss: D1, D3 (78 of 105 tokens)
skip what does not fit: D1, D3, D9 (102 of 105 tokens)The skimming reader is a toy that looks only at the first and last passage. Real models are not that extreme, but position can change what they use.
Expected output
best first D8 D3 D1 D6 D4
the skimming reader sees: D8 and D4
best at the edges D8 D1 D4 D6 D3
the skimming reader sees: D8 and D3Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the overhead
In the Python tab of the packing example, change OVERHEAD = 8 to OVERHEAD = 0.
Myth vs fact
Myth
If the window is big enough, send everything retrieved.
Fact
Cost and latency grow with every token, and extra passages can bury the right one. Send what your evaluation shows helps.
Myth
Tokens are the same as words.
Fact
A token is often a piece of a word. English averages about four characters per token, but numbers, code and other languages can be far denser.
Myth
The best passage should always be first.
Fact
It is a good default. Some models use the end of a long prompt well too, so test best-at-both-ends if you pass many passages.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Reserve the answer's tokens as well as the prompt's. If the window is the sum of input and output, a prompt that fills it leaves no room to answer and the reply is cut off.
- ▸
Cost is linear in input tokens. With an example price of 0.50 dollars per million input tokens, a 1,500-token prompt costs about 0.00075 dollars, so 100,000 questions a day cost about 75 dollars. Halving the sources halves most of that (Topic 11.5 works this out properly).
- ▸
Prompt caching rewards a stable prefix. Put the instructions first and the changing sources later, and many providers can reuse the cached prefix. Check your provider's rules before relying on it.
Remember this
- 1
Why a budget. A model reads a fixed-size window (LLM Engineering Topic 1.5), and everything in it is paid for in money, time and attention: the instructions, every source, the question, and the room left for the answer. Passing every retrieved passage 'to be safe' costs more, answers slower, and often answers worse because the right passage is surrounded by wrong ones.
- 2
Counting tokens by hand. A token is the unit a model reads, roughly a short word or a piece of a word. Real counts need the model's own tokenizer. For planning, a common rule of thumb for English is about one token per four characters, and this course uses
(characters + 3) // 4so the answer is a whole number. It is a rough rule: code, numbers and other languages differ, and you should leave a margin or use the provider's counter for the final check. - 3
What the budget covers. Sources get a share of the window, not the whole of it. Reserve room for the instructions, the question, the answer you want back, and the tags around each source (the example charges eight tokens of overhead per passage). If a model's window is large it is tempting to fill it; resist, because cost and latency grow with every token you send.
- 4
Packing best first.
retrievereturns passages in rank order. Walk down the list and keep each passage while the running total stays inside the budget. The one real decision is what to do when a passage does not fit: stop (never use a lower-ranked passage in place of a higher one) or skip (a shorter, lower-ranked one may still fit). In the example, stopping keeps two passages and skipping keeps a third. Skipping uses the budget better, stopping keeps the rank order honest. Pick one on purpose and write it down. - 5
Order matters too. Models read the whole prompt, but not evenly. A 2023 study ('Lost in the Middle', Liu and others, published in TACL) found that on long inputs models answered best when the relevant passage was at the start or the end, and worse when it sat in the middle. It tested question answering over many documents and a key-value lookup, on the models of that time. The effect depends on the model and the length, so treat it as a reason to test, not a rule. A common choice is best first; the alternative is best at both ends, with the weakest passages in the middle. The second example prints both orders for five ranked passages.
- 6
More context is not better. Past a point extra passages add noise: a near-duplicate, a passage about a different product, an old version. Choose
kfrom your evaluation (Phase 9), not from how much fits. Often three well-chosen passages beat ten.
Explain it without notes
How do you estimate tokens without a tokenizer, and when is that not good enough?
Describe the 'stop' and 'skip' packing policies and the trade-off between them.
Why is sending more passages not always better?
What can the order of passages change?
Practice
Write pack(hits, budget, overhead) where hits is a list of (doc_id, text) pairs in rank order. Keep a passage only when its cost (estimate plus overhead) still fits; skip passages that do not fit and carry on. Return the kept ids. Print the result for the example hits with a budget of 80.
Write at_the_edges(ranked) that puts rank 1 first and rank 2 last, rank 3 second and rank 4 second to last, and so on. Print the order for ranks ["a", "b", "c", "d", "e", "f"].
Trade-offs
- ↔
Small budget: cheap and fast, but a relevant passage can be cut. Large budget: safer for recall, costlier, and adds noise.
- ↔
Stop policy: honest rank order. Skip policy: fuller use of the budget but a lower-ranked passage can replace a higher one.
- ↔
Best-first order is simple to explain. Best-at-both-ends can use long prompts better but is harder to read when debugging.
Done when you can
Done when you can estimate tokens with a stated rule and say when it is not enough.
Done when you can pack passages best-first into a budget with overhead.
Done when you can explain the stop and skip policies.
Done when you can arrange passages best-first or best-at-both-ends and say why you chose one.