Topic 13.2
Design: Question Answering over Company Documents
In one line
A worked system-design answer for 'build question answering over our company documents': requirements, back-of-envelope numbers, ingestion, retrieval, answers, evaluation, security and scale, with the arithmetic done by hand.
Think of it like this
Designing a library's help desk. Before choosing shelves you ask how many books there are, who may borrow which ones, how fast an answer must come and what happens when the librarian is wrong. The shelves are the last decision. A design answer works the same way: requirements and numbers first, technology last.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Back-of-envelope estimate
- A quick calculation with stated assumptions, to see whether a design is feasible.
- Ingestion path
- The offline pipeline that turns documents into searchable chunks.
- Query path
- The online steps from a question to a cited answer.
- Access label
- A field on each chunk saying who may see it, checked at search time.
- Peak factor
- How many times busier the busiest moment is than the average.
- Golden set
- Real questions with the passages that answer them, used to measure quality.
Step by step
01Step 1: write the requirements down
Documents, size, change rate, questions per day, peak, latency, citations, access rules, cost. Say the assumptions aloud. If the interviewer corrects one, your design adapts.
02Step 2: do the arithmetic
Chunks, vector bytes, index overhead, embedding tokens, questions per second, tokens and dollars per question. The estimator prints each. The point is not precision, it is knowing whether the answer is 'one machine' or 'a cluster'.
03Step 3: draw both paths
Draw ingestion on the left and query on the right, joined by the index. Mark which parts are batch and which are online.
04Step 4: justify retrieval choices
Hybrid because handbooks mix exact terms and paraphrases (Topic 6.1). A vector index with a filter-aware design because every query has a tenant or team filter (Topic 3.7). A re-ranker only if the golden set shows a gain.
05Step 5: define the refusal and the citation rules
A threshold before the model call, one fixed refusal sentence, and a verifier that rejects any citation that was not retrieved. These are cheap, testable and what separates a product from a demo.
06Step 6: evaluation, security, operations
Golden set and CI gates. Access labels, untrusted-text handling, redaction and deletion. Dashboards for latency, refusal rate and cost; alerts when they move. Finish with the rollout: shadow mode, a small share of users, then everyone, with a kill switch.
Try it yourself
- 1
Scale the corpus
Set DOCS to 2,000,000 and run. Do the vectors still fit in one machine's memory? Which line changes most, and which stays the same?
- 2
Change the model
Set DIMENSIONS to 1536. How does the memory change? Does the cost per question change? Explain why one changes and the other does not.
- 3
Cut the cost
Find the biggest lever on the cost line. Try SOURCES of 3 by editing the calls to cost_per_question, then reason about what you risk by sending fewer passages.
Code & diagrams
Prices and the index overhead are examples and assumptions. Replace them with your provider's numbers and a measured overhead for your index.
Expected output
chunks: 200000
vectors: 307.2 MB raw, 460.8 MB with index overhead
embedding the corpus once: 60000000 tokens, $1.20
load: 0.23 questions/s average, 1.16 at peak, about 4.6 in flight at peak
prompt: 1700 tokens in, 260 out
cost: $1.24 per 1000 questions, $24.80 a day, $744.00 a month
input share of the cost: 68.5%
with 3 sources: $0.94 per 1000 questions, 24.2% cheaperBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget that vectors are 4 bytes each
In the Python tab, change chunks * DIMENSIONS * 4 / 1e6 to chunks * DIMENSIONS / 1e6 and run.
Myth vs fact
Myth
A design answer is a list of technologies.
Fact
It is requirements, numbers and reasons. Technologies are the last line, and each needs a reason that comes from the numbers.
Myth
Vector memory is the main cost of RAG.
Fact
At this scale the vectors fit on one machine. The recurring cost is the model reading retrieved passages on every question, which the estimator shows is most of the bill.
Myth
Security is a separate topic from design.
Fact
Access labels shape the index schema and the search filter. Added late, they force a re-ingestion.
Interview problem
The problem
Question answering over 20,000 company documents
Design a service that lets employees ask questions in plain language and get answers with sources from the company's documents (policies, runbooks, wiki pages). Some documents are visible only to certain teams.
You're given
- About 20,000 documents, around 10 chunks each, edited daily.
- 20,000 questions a day, peak five times the average, answers within about 6 seconds.
- Every answer cites its sources. The assistant says it does not know when the documents have no answer.
- Team-restricted documents must never reach someone outside the team, including through logs or caches.
- A budget of about 1,000 dollars a month for model calls.
The interviewer follows up
A document was edited an hour ago. The assistant still quotes the old version. Why?
How do you handle 10 times the traffic?
How do you know a new embedding model is better?
Someone asks for data they should not see. What stops them?
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
State your assumptions as variables (peak factor, seconds per request, index overhead). Interviewers rarely check the digits but always check that you know which numbers are guesses.
- ▸
Latency is a sum: embed the question, search, re-rank, build, generate. Time to first token is dominated by the model's input length, so fewer, shorter passages help twice: cost and speed (Topic 11.4).
- ▸
Mention the change-management path: switching the embedding model means a second index, a backfill and a measured cut-over, because old and new vectors cannot be mixed (Topic 11.8).
Remember this
- 1
Why practise a full design. Design questions combine everything: ingestion, search, generation, evaluation and security. Interviewers watch whether you remember the parts nobody asked about. The usual gaps are access control, evaluation and what happens when a document changes.
- 2
Requirements to extract. Ask: how many documents and how large, how often they change, who asks and how many questions a day, what a wrong answer costs, who may see which document, what languages, and whether answers need citations. Write the numbers down. In the worked example: 20,000 documents, about 10 chunks each, 20,000 questions a day, a peak five times the average, citations required, some documents restricted by team.
- 3
Estimates, done by hand. 20,000 documents times 10 chunks is 200,000 chunks. At 384 dimensions and 4 bytes each, one vector is 1,536 bytes, so the vectors take 307.2 MB; a graph index adds more on top. Half again is a generous allowance (a graph with 16 links per vector adds nearer 10 percent, but state the assumption and measure yours), about 460.8 MB, which fits in memory on one machine. Embedding the corpus once is 200,000 times 300 tokens, 60 million tokens, a one-off cost of a dollar or two at example prices. The runnable estimator prints all of this, plus load and cost per question.
- 4
The two paths. The ingestion path (offline): load, clean, chunk with context, attach metadata including an access label, embed, upsert with stable ids (Phase 4). The query path (online): rewrite or standalone-ify the question if needed (Phase 7), search keyword and vectors with the access filter applied before ranking, fuse with reciprocal rank fusion, optionally re-rank, check the evidence bar, build a numbered prompt, generate, verify citations, return the answer with sources (Phases 6 and 8).
- 5
Evaluation is part of the design. Say that you will build a golden set before you tune anything: real questions with the passages that answer them, plus unanswerable ones. Track hit rate, MRR, refusal rate and citation validity, and gate changes in CI (Phase 9). Without this the interviewer hears a system nobody can improve safely.
- 6
Security and scale in one breath. Access labels on every chunk and a filter before the model sees text (Topic 10.3); retrieved text treated as data, never as instructions (Topic 10.2); personal data redacted before indexing (Topic 10.5); deletion that reaches vectors, caches and logs (Topic 10.6). Scale: 460 MB of vectors fits on one node, so shard only when the numbers say so; cache embeddings and popular answers (Topic 11.3); budget latency per stage (Topic 11.4).
- 7
Close with what you would do first. Ship a keyword baseline with a golden set in the first week, add vectors and hybrid fusion next, and add re-ranking only if the metrics show a gain (Topic 6.7). It tells the interviewer you know that measurement, not the fanciest stack, decides.
Explain it without notes
Walk through the arithmetic for 20,000 documents of 10 chunks each at 384 dimensions.
Why is access control part of the data model and not a final step?
What do you build first, and why before any clever retrieval?
Where does the money go in a RAG question, and what is the biggest lever?
Practice
A store holds 1,000,000 chunks embedded at 768 dimensions as 32-bit floats. Print its raw vector size in gigabytes with three decimals.
The same vectors are stored as one byte per number (8-bit). Print both sizes, and the size as a percentage of the float version.
Trade-offs
- ↔
One node versus a cluster: one node is simpler to run and enough at this size, a cluster adds failure modes before the numbers demand it.
- ↔
Hybrid search versus vectors only: hybrid handles exact identifiers and paraphrases, at the price of two retrievers to run and tune.
- ↔
Citations and refusals versus a smoother chat feel: they cost a few percent of answered questions and buy trust and testability.
Done when you can
Done when you can estimate chunks, vector bytes and cost per question from a short requirement list.
Done when you can draw the ingestion and query paths from memory.
Done when you can say where access control, evaluation and deletion sit in the design.
Done when you can state your top three assumptions and how you would check them.