Topic 14.3
System Design: Document Q&A at Scale
In one line
Design a service that answers questions from millions of private documents for many companies, with ingestion, hybrid retrieval, citations, freshness, deletes and strict separation between tenants. The estimator gives storage, ingestion cost and question cost.
Think of it like this
One big library shared by twenty companies. Each company has locked rooms. A good librarian finds the right page fast, tells you which book and page the answer came from, and never walks into another company's room. When a book is updated or thrown away, the old copy must disappear from every shelf and every card in the catalogue.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Ingestion
- Loading documents into the system: reading, cleaning, splitting, embedding and storing them.
- Chunk overlap
- A few tokens repeated between neighbouring chunks, so a sentence cut at the edge still appears whole in one chunk.
- Metadata filter
- A condition on stored fields, such as tenant id equals 7, applied while searching.
- Rerank
- Take a wide set of search hits and reorder them with a slower, more accurate scorer to keep the best few.
- Idempotent
- Safe to run twice: the second run gives the same result as the first.
- Tombstone
- A marker saying a record is deleted, so searches skip it until a cleanup job removes it.
- Recall at k
- The share of questions where the right chunk is somewhere in the top k search results.
- Embedding
- A list of numbers that stands for the meaning of a piece of text, so that similar meanings are close together. See Topic 1.3.
- Hybrid search
- Running keyword search and vector search together and merging their results. See Topics 8.2 and 8.3.
- Citation
- A marker in the answer, such as [3], that points to the chunk the claim came from. See Topic 8.6.
Step by step
01Step 1: fix the requirements with numbers
Write them down: 20 tenants, 2 million documents of about 3,000 tokens, 100,000 questions a day, 5 seconds at p95, edits searchable within 10 minutes, deletes immediate, answers cite the document and page, and a monthly budget. Ask about permissions at once: per tenant is simple, per document is harder.
02Step 2: estimate storage and cost
With 400-token chunks and 50 tokens of overlap, a 3,000-token document makes 9 chunks, so 18 million chunks. At 1,024 dimensions of 4 bytes each, vectors take about 74 GB raw and about 110 GB with an index (a rule of thumb: measure your own). Run the estimator to see the first-load embedding cost, the daily re-embedding of changed documents, and the cost of a question.
03Step 3: draw both paths
Draw ingestion on top and the query path below, joined by the index. Say what is asynchronous and what is not. The diagram shows one layout.
04Step 4: permissions and tenants
Every chunk carries the tenant id and, if needed, the groups allowed to read it. The API takes the tenant and groups from the signed-in session, never from the request body, and passes them as a filter to the search. Test with a second tenant on purpose.
05Step 5: freshness, deletes and model changes
A queue event for every change; version numbers; upsert new chunks, then retire old ones; tombstones for deletes; a dashboard of ingestion lag. For a new embedding model, build a second index in the background, compare recall on a test set, then switch the alias and delete the old index.
06Step 6: evaluate and operate
Measure retrieval first (recall at k on questions with known source chunks), then the answer (is every claim supported by a cited chunk, does it refuse when nothing is found). Watch p95 latency, time to first token, lag, cost per 1,000 questions and the share of answers that refused.
Try it yourself
- 1
Change the chunk size
Set
CHUNK_TOKENSto 200 andOVERLAP_TOKENSto 20. How many chunks does one document make, and what happens to vector storage? Why might smaller chunks improve precision but cost more to store? - 2
Use half-size vectors
Set
BYTES_PER_NUMBERto 2 (float16). By how much does storage fall? Name one risk of storing lower-precision vectors and how you would test for it. - 3
Trace a delete
Write the steps, on paper, from a user deleting a file to the moment it can no longer appear in an answer. Which steps must be synchronous and which can be background work?
Code & diagrams
Example sizes and prices. The index overhead of 1.5 is a rule of thumb, so measure it for the store you choose.
Expected output
chunks per document: 9, chunks in total: 18000000
vectors: 73.7 GB raw, 110.6 GB with index; chunk text: 28.8 GB
embedding cost: first load 720 dollars, then 14 dollars per day for changed documents
one question: 2520 tokens in, 250 out, 1.635 dollars per 1000 questions
questions cost per month: 4905 dollars
peak questions per second: 6.94, vector lookups at peak: 34.7 chunks per secondBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Mix up the units
In per_question, drop the division by one million for the input tokens: input_per_question * PRICE_IN instead of input_per_question / 1e6 * PRICE_IN (Python).
Myth vs fact
Myth
Vector search alone is enough.
Fact
Names, codes and rare words are often found better by keyword search. A hybrid of both, with a rerank step, is the usual strong baseline.
Myth
Put the permission check in the prompt.
Fact
A prompt is a request, not a lock. Enforce permissions in the data layer so the model never sees what the user may not read.
Myth
Ingestion is a one-off job.
Fact
Documents change daily. Ingestion is a running service with queues, retries, versions and monitoring.
Interview problem
The problem
Document Q&A for many companies
Design a service where each company uploads its documents (PDFs, wiki pages, tickets) and its employees ask questions. Answers must cite the source and respect who is allowed to read what.
You're given
- 20 tenants, 2 million documents of about 3,000 tokens each.
- 100,000 questions a day, peak 6 times the average, 5 seconds at p95.
- Edits searchable within 10 minutes. Deleted documents disappear from answers immediately.
- Answers cite document and page, and say "not in the documents" when nothing relevant is found.
- No tenant may ever see another tenant's data. Some documents are restricted to groups.
- Model and embedding budget around 5,000 dollars a month at example prices (check your provider).
The interviewer follows up
How do you prove tenant A cannot see tenant B's data?
A document is deleted. How fast must it disappear, and how do you make that true?
Answers are fluent but cite the wrong page. Where do you look?
A user is removed from a group, or a document becomes restricted. How fast does the search notice?
You want a better embedding model. What is the plan?
When it breaks
Permissions are applied after the search returns its top 50
What you see
Users see empty or poor answers when most hits belong to others, and one forgotten check leaks data.
Fix & prevent
Put the tenant and group filter inside the search query and test with a second tenant.
A document is updated but old chunks stay in the index
What you see
Answers quote outdated policy next to the new one.
Fix & prevent
Version documents, upsert new chunks then retire old ones, and alert on ingestion lag.
The embedding model is swapped without re-embedding
What you see
Search quality collapses because query and stored vectors come from different models.
Fix & prevent
Store the model id with each vector, refuse mixed searches and migrate through a second index.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Batch embedding calls (many chunks per request) and respect the provider's tokens-per-minute limit; a first load of millions of chunks is a rate-limit problem as much as a cost problem (Topic 6.7).
- ▸
Rerank widens recall cheaply: retrieving 50 and keeping 5 usually beats retrieving 5 directly. Measure recall at 50 and precision at 5 separately.
- ▸
The estimate prices embeddings and answers only. Vector database hosting, the reranker, parsing (OCR for scanned pages is often the biggest ingestion cost) and evaluation runs come on top, so state that in the interview. At peak the model sees about 1 million input tokens a minute (6.94 questions a second x 60 x 2,520), which you compare with the provider's tier.
- ▸
Store a hash of each chunk's text. If an edit does not change a chunk, skip the embedding call: for a 2% daily change this keeps the daily bill close to the 14 dollars in the example.
Remember this
- 1
Why this exists. Asking questions about your own documents is the most common LLM product. A script that works on 50 files stops working at 2 million: loading takes days, files change every hour, people may only see some documents, and the bill and the index grow. At scale the hard parts are ingestion, permissions, freshness and cost, not the model call.
- 2
Two paths, drawn separately. The ingestion path runs in the background: sources go into a queue, workers parse, clean and chunk, an embedder turns chunks into vectors in batches, and the results are written to the index. The query path runs online under a latency limit: authenticate, search with filters, rerank, build the prompt with numbered chunks, call the model, check the citations. Keeping them apart lets you scale, retry and budget each one alone.
- 3
Ingestion details. Parsing is the quiet source of bad answers: tables, scanned pages, headers and footers repeated on every page. Chunk by structure (headings, paragraphs) with a small overlap, keep a chunk size that suits your questions (400 tokens in the example), and attach metadata: tenant id, document id, version, page, date and permissions. Make each step idempotent with a content hash, so a retry or a re-run never creates duplicates. Failed documents go to a dead-letter queue that someone looks at.
- 4
Retrieval details. Combine keyword search (good for names and codes) with vector search (good for paraphrases), apply the tenant and permission filter inside the search, not after it, take the top fifty, rerank to the best five, and put them in the prompt with ids. Filtering after the search can return nothing useful (all fifty hits belonged to others) and is a leak risk if you ever forget it.
- 5
Freshness and deletes. Users expect edits within minutes and deletes at once. Store a version per document, upsert new chunks before removing old ones, and mark deleted documents with a tombstone that the search filter respects until a cleanup job removes them. Store the embedding model id with each vector: vectors from different models cannot be compared, so changing the model means re-embedding everything into a new index and switching over.
- 6
Multi-tenancy. Put a tenant id on every row and enforce it in the data layer, not in prompts. Add per-tenant quotas so a noisy tenant cannot use all the capacity (Topic 12.5), and consider a separate index or database for very large or regulated tenants. Write a test that asks tenant A about tenant B's secrets and expects a refusal.
Explain it without notes
Why do you keep ingestion and the query path separate?
Why must the permission filter be applied inside the search, not after it?
How do you handle a document update and a delete?
How would you evaluate retrieval separately from the final answer?
Practice
Write chunks_for(doc_tokens, chunk, overlap) that returns how many chunks a document makes (1 if it fits in one chunk). Print the result for documents of 300, 400, 401, 1,000 and 3,000 tokens with chunk 400 and overlap 50.
Write vector_gb(chunks, dimensions, bytes_per_number=4) and print the storage of 18 million chunks with 1,024 dimensions in float32 and in float16, and with 384 dimensions in float32.
Trade-offs
- ↔
Small chunks give precise matches but more vectors and less context per chunk; large chunks do the reverse. Pick with recall tests.
- ↔
Reranking improves quality but adds latency and cost per question; use it when the eval shows a gain.
- ↔
One shared index is cheapest to run, separate indexes isolate tenants best; many designs share by default and isolate the largest or most regulated tenants.
Done when you can
Done when you can draw the ingestion path and the query path and say what is asynchronous.
Done when you can estimate chunks, vector storage, embedding cost and question cost.
Done when you can explain where tenant and permission filters are enforced and how you test them.
Done when you can describe updates, deletes and an embedding model change.
Done when you can name two retrieval metrics and two answer metrics.