Topic 13.5
Trade-off Questions
In one line
How to argue a choice instead of reciting a preference: chunk size, k, hybrid or not, re-rank or not, which store, RAG or long context. Each is a table of costs and a measurement.
Think of it like this
Choosing between a bus, a bike and a taxi. None wins everywhere. The bus is cheap and slow, the taxi fast and costly, the bike free and tiring in rain. Nobody says 'bikes are best'. They say 'it depends on distance, weather and budget, and here is what I would pick for this trip'. A trade-off answer has the same shape.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Trade-off
- Getting more of one good thing by accepting less of another.
- Criterion
- A property you judge options by: precision, cost, latency, effort.
- Weight
- How much a criterion matters for this product, as a number.
- Diminishing returns
- Each extra unit helps less than the one before.
- Decision rule
- A stated condition that makes the choice, such as 'stop when the next passage adds under 5 points'.
- Reversible decision
- A choice that is cheap to change later, so you can decide quickly.
Step by step
01Step 1: list the options
Write two or three real options. 'Small, medium and large chunks' is better than 'chunks versus no chunks'.
02Step 2: choose the criteria
Pick three to five that matter for this product. A legal search cares about completeness, a mobile assistant about latency and cost.
03Step 3: score each option
Give each option a score per criterion, from measurements where you have them and judgement where you do not. Say which are measured.
04Step 4: weight and decide
Weights depend on the product. Change them and the winner may change, which is exactly the point: the right answer depends on what you value.
05Step 5: find the stopping point for numbers like k
For a numeric knob, compute cost and quality for each value and stop where the next step adds less than a gain you set in advance.
06Step 6: say what would change your mind
End with a measurement: 'if hit rate at k=3 falls under 80 percent on the golden set, I would add a re-ranker'. It shows the decision is revisable.
Try it yourself
- 1
Make a third profile
Add a profile 'completeness only' with weights 0, 5, 0. Which chunk size wins? What does that say about asking for the single best choice without saying what matters?
- 2
Change MIN_GAIN
Set MIN_GAIN to 2 and run the second example. How does the pick move? What is the extra cost of that choice per 1000 questions?
- 3
Bring your own trade-off
Replace the options with 'BM25', 'vectors' and 'hybrid', and the criteria with 'exact terms', 'paraphrases' and 'operations effort'. Score them and decide for a product of your choice.
Code & diagrams
The scores are example judgements. In practice take precision and completeness from the golden set and cost from the estimator.
Expected output
accuracy first (precision x2, completeness x2, cost x1)
medium (300 tokens) 19
small (100 tokens) 18
large (800 tokens) 15
pick: medium (300 tokens)
budget first (precision x1, completeness x1, cost x3)
small (100 tokens) 19
medium (300 tokens) 17
large (800 tokens) 10
pick: small (100 tokens)The hit rates are example numbers. Measure your own with the harness of Topic 9.9. Prices are examples: check your provider.
Expected output
k=1 hit 55% tokens 500 $0.64 per 1000 gain +55
k=2 hit 72% tokens 800 $0.79 per 1000 gain +17
k=3 hit 81% tokens 1100 $0.94 per 1000 gain +9
k=4 hit 85% tokens 1400 $1.09 per 1000 gain +4
k=5 hit 86% tokens 1700 $1.24 per 1000 gain +1
k=6 hit 86% tokens 2000 $1.39 per 1000 gain +0
pick: k=3 (the next passage adds under 5 points)Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Weight everything the same
In the first example's Python tab, set both profiles' weights to [1, 1, 1] and run.
Myth vs fact
Myth
There is a best chunk size.
Fact
There is a best size for a corpus, a question mix and a budget. You find it by measuring several sizes on your own golden set.
Myth
More retrieved passages always improves answers.
Fact
Past a point extra passages add cost and distraction, and in the example the sixth passage adds nothing.
Myth
A bigger or newer component is the safe choice.
Fact
It adds latency, cost and operations. Choose it when a measurement shows the gain.
Interview problem
The problem
Chunk size, k and re-ranking for a policy search
An HR team wants answers from long policy documents. Quality matters more than cost, latency should stay under 5 seconds, and the team is two engineers. Choose chunk size, k and whether to add a re-ranker, and defend it.
You're given
- About 3,000 policy pages with headings, edited monthly.
- Questions are specific ('how many days of carer's leave after one year?').
- A wrong answer about a policy is costly. Latency must stay under 5 seconds at p95.
- Two engineers to build and run it.
The interviewer follows up
Latency is now 7 seconds at p95. What do you cut first?
The interviewer says money is no object. Does your design change?
Why not just paste all 3,000 pages into a long context?
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prefer reversible decisions made fast over irreversible ones made slowly. The store behind an interface and a re-embeddable index are reversible. A schema without access labels is not.
- ▸
Report trade-offs in the currency of the business: dollars per 1000 questions, seconds at p95, percentage points of hit rate, engineer-days a month to operate.
- ▸
Interviewers often change a constraint mid-answer ('now latency must be under 2 seconds'). Redo the table with new weights out loud. It is the best sign of understanding.
Remember this
- 1
Why interviewers ask these. They want to see whether you know what each option costs and whether you would decide from evidence. A candidate who says 'hybrid is best' has a preference. A candidate who says 'hybrid costs a second retriever and a fusion step, and pays off when questions mix exact identifiers and paraphrases, which I would check on a golden set' has judgement.
- 2
A four-part answer. (1) Name the options. (2) Name the criteria that matter for this product, such as precision, completeness, cost, latency, operations burden. (3) Say which option wins on which criterion. (4) Decide for this situation and say what measurement would change your mind. The trade-off table in the example turns this into a calculation you can show.
- 3
Chunk size (Topic 4.4). Small chunks give precise matches and cheap prompts, but lose context. Large chunks keep context but dilute the match and cost tokens. Typical answer: start around a few hundred tokens with structure-aware boundaries and heading context (Topic 4.5), then measure hit rate at several sizes, or use parent-child retrieval to search small and read large (Topic 12.1).
- 4
k, the number of passages (Topic 8.2). More passages raise the chance the right one is included and cost tokens, latency and distraction. Choose the smallest k whose next step adds little, as in the second example.
- 5
Hybrid or not, re-rank or not (Topics 6.1, 6.4, 6.7). Hybrid adds a second retriever and fusion for robustness to exact terms and paraphrases. A re-ranker adds a model call per query and latency for better ordering of a shortlist. Both are justified by measured gain on your own questions, and skipped when plain BM25 or plain vectors already do well.
- 6
Which store (Topic 5.8). A vector extension in the database you already run is often enough and has one system to operate. A dedicated vector database earns its place at larger scale, with heavy filtering or specialised features. Hide the choice behind an interface so you can change it later.
- 7
RAG or long context (Topic 12.8). Pasting everything wins for a small, stable corpus and rare questions: no retrieval failure, simple. RAG wins when the corpus is large, changes, needs access control or citations, or when per-question cost matters. Using both, with retrieval narrowing a long context, is common.
Explain it without notes
Give the four-part structure of a trade-off answer.
How would you choose a chunk size and defend it?
How do you decide between hybrid search and vectors only?
When is long context a better answer than RAG?
Practice
Option A scores precision 5, cost 2. Option B scores precision 3, cost 5. With weights precision 2 and cost 1, print the weighted total of each and the pick.
A team sends 5 passages of 300 tokens per question and considers 3. For 20,000 questions a day at an input price of 0.50 dollars per million tokens, print the dollars saved per day.
Trade-offs
- ↔
Deciding by measurement: slower to start, defensible and revisable.
- ↔
Deciding by experience: fast, right more often than not, and hard to explain to a sceptic.
- ↔
A scoring table: forces criteria into the open, but the scores are only as good as the evidence behind them.
Done when you can
Done when you can give the four-part structure and apply it to chunk size.
Done when you can compute a stopping point for k from cost and hit rate.
Done when you can say what measurement would change each of your choices.
Done when you can redo a trade-off out loud after a changed constraint.