Topic 11.1
Architecture of a RAG Service
In one line
A RAG service is two programs that share data: an ingestion path that turns documents into an index when they change, and a query path that answers a question in a few seconds. Knowing which parts are online, which are batch and where state lives is the first production decision.
Think of it like this
A restaurant has a kitchen prep team and a service team. Prep chops, marinates and labels ingredients in the morning, without any customer waiting. Service takes an order and plates a dish in minutes, using what prep left on the shelves. If prep is late, service has nothing fresh; if service has to chop, customers wait. A RAG service has the same two crews: ingestion prepares the shelves, the query path serves the order.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Ingestion path
- The part of the system that turns changed documents into index entries. Slow work, no user waiting.
- Query path
- The part that answers one question: embed, search, re-rank, prompt, generate. A user waits for it.
- Source of truth
- The original data (the documents). Everything else is derived from it.
- Derived state
- Data computed from the source of truth, such as the index. It can be rebuilt.
- Idempotent
- Safe to run twice: the second run changes nothing. It makes retries harmless.
- Index version
- A counter that changes whenever the index contents change.
- Throughput
- How much work is finished per hour. Ingestion cares about this.
- Latency
- How long one request takes from start to finish. The query path cares about this.
Step by step
01Step 1: draw the two paths
Put the documents on the left, the user on the right and the index in the middle. Everything to the left of the index is ingestion, everything to the right is the query path. The index is the only thing both sides touch.
02Step 2: decide what is the source of truth
Write down, for each piece of data, who owns it and whether it can be rebuilt. Documents: owned by the wiki. Index: derived, rebuildable. Cache: disposable. The example keeps a STORE (truth) and an INDEX (derived) as two separate objects for that reason.
03Step 3: make ingestion idempotent
Key every chunk by (doc_id, chunk_number, doc_version). Writing the same key twice replaces, not duplicates. Then a crashed ingestion job can simply be run again. In the snippet, index stands for any store with a delete-by-document call and a put call (Phase 5 builds a real VectorStore): the document's old chunks go first, then the new ones are written.
def upsert(index, doc_id, version, chunks):
index.delete_where(doc_id=doc_id) # remove the old chunks of this document
for n, chunk in enumerate(chunks):
index.put(key=(doc_id, n, version), text=chunk)04Step 4: keep the query path read-only
The query path should never write to the index. Then you can run ten copies of it, restart them any time and scale them separately. Writes (feedback, logs) go to a different place.
05Step 5: connect them with a version, not with a lock
Do not make a question wait for an ingestion job. Let ingestion finish, bump the index version, and let the next question see the new state. The example prints the version after every ingest call so you can watch it move.
06Step 6: handle deletes on purpose
An ingestion path that only adds will leak removed documents (a withdrawn policy) into answers forever. Treat "document deleted" as an event that removes every chunk of that document and bumps the version.
Try it yourself
- 1
Add a document
Predict first: add
ingest("D4", "A helmet is included free with every rental. Riders under 16 must wear one.")before the questions. Which version number does the last line show? Run to check. - 2
Break the split
Make
querycallingestwhen it finds no answer. What would a user wait for? This is the "batch work on the online path" mistake from the lesson. - 3
Compare languages
Run the Python, JavaScript and Java tabs. Same lines, same version numbers. Find where each language keeps the "derived" index.
Code & diagrams
A toy keyword index so that three languages print the same text. The structure is the point: ingest() writes, query() only reads, and the version moves on every write.
Expected output
ingest D3: 19 distinct words, index version 1
ingest D8: 15 distinct words, index version 2
ingest D2: 16 distinct words, index version 3
ask: What does the monthly pass cost? -> The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
ask: Do you rent scooters? -> I can't find that in the handbook.
ingest D8: 15 distinct words, index version 4
ask: What does the monthly pass cost? -> The monthly pass costs 30 dollars and includes unlimited rides of up to 45 minutes. [D8]
store: 3 documents, index: 3 entries, version 4Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to bump the version
In the Python tab, delete the line STATE["index_version"] += 1 and run.
Myth vs fact
Myth
A RAG service is a chain: one function that calls the next.
Fact
That is the query path only. The ingestion path is a separate system with its own failures, queues and deploys, and most production incidents start there.
Myth
The vector database is the database of record.
Fact
It is a derived index. Keep the documents and their versions somewhere you can rebuild from; otherwise a bad migration cannot be undone.
Myth
Indexing can run the first time a document is needed.
Fact
Embedding a large document takes far longer than any user will wait. Do it ahead of time, and show what is not indexed yet as not available.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Separate deploy units: the query service and the ingestion workers have different scaling signals (requests per second versus queue length) and different failure costs. A bad ingestion deploy should not take answers down.
- ▸
Put a queue between "document changed" and "index updated". It absorbs bursts (a bulk upload), allows retries with backoff and gives you a number to watch: the backlog. Topic 11.7 returns to the backlog.
- ▸
Store the embedding model name and chunking settings with every index version. Without them, nobody can say which settings produced the vectors a bad answer came from (Topic 11.8 depends on this).
Remember this
- 1
Why this split exists. The two paths want opposite things. Ingestion is throughput-bound: millions of passages, slow model calls for embeddings, retries, it may take hours and nobody is waiting. The query path is latency-bound: one question, a few seconds, a person watching. If you build them as one program, a big re-index slows every answer, and a slow answer path forces you to cut corners in ingestion. Splitting them lets each scale, fail and be deployed on its own.
- 2
The ingestion path (mostly batch or event-driven): fetch a changed document, parse it, clean it, chunk it (Phase 4), attach metadata (owner, permissions, source URL, version), embed the chunks (Phase 2), and write them to the index (Phase 5). It also handles deletes: a removed document must disappear from the index, not only stop being added. It should be idempotent: running it twice on the same document leaves the same index, so a retry never creates duplicates.
- 3
The query path (online): normalise the question, rewrite it if needed (Phase 7), embed it, search (Phases 3, 6), re-rank, build the prompt, call the model, check the answer (Phase 8), return the answer with sources. It is a pipeline of stages, and each stage will get its own cache (Topic 11.3), budget (Topic 11.4) and trace span (Topic 11.6).
- 4
Where state lives. Three kinds. Source of truth: the original documents and their versions, owned by another system (a wiki, a file store). Derived state: the index, built from the source of truth, which can be thrown away and rebuilt. Disposable state: caches and queues. Production rule: never let the index be the only copy of anything. If you can rebuild it from the source of truth, a corrupted index is a bad afternoon, not a disaster.
- 5
What is online and what is batch. Online (the user waits): query embedding, search, re-rank, generation. Batch or background: parsing, chunking, document embedding, index build, backfills (Topic 11.8), evaluation runs (Phase 9). A common mistake is putting a batch task on the online path, such as embedding every document the first time anyone asks about it. The first user then waits minutes.
- 6
The index is shared state, and a version number makes that safe. Both paths touch the index, so give it a version that changes whenever its contents change. The query path can then record which version it answered from (traces, Topic 11.6), and caches can include it in their keys (Topic 11.3). The example below keeps a tiny version counter for exactly this. LLM Engineering Phase 8 sketched RAG as a short pipeline; this topic is the same pipeline seen as two deployable systems.
Explain it without notes
Why split a RAG service into an ingestion path and a query path?
What is the source of truth in a RAG system, and why does it matter that the index is derived?
What does it mean for ingestion to be idempotent, and how do you achieve it?
Name three things that must never happen on the online query path.
Practice
Write upsert(index, doc_id, chunks) for an index that is a dict from (doc_id, n) to text. It must remove all old entries of doc_id first, then add the new chunks, and return the number of entries removed. Call it twice with the same chunks and show the index size does not change.
Write a function deleted(index, doc_id) that removes every entry of a document and returns how many it removed. Show a query that finds the document before and does not after.
Trade-offs
- ↔
Two deploy units: independent scaling and safer deploys, but two things to operate and a contract (the index version) to keep.
- ↔
Event-driven ingestion: fresh index and a natural backlog metric, but more moving parts than a nightly job (Topic 11.2).
- ↔
Rebuildable index: easy recovery, but a rebuild costs time and embedding money (Topic 11.5).
Done when you can
Done when you can draw the ingestion path and the query path and say what each stage does.
Done when you can name the source of truth, the derived state and the disposable state in a RAG system.
Done when you can explain why ingestion must be idempotent and how to make it so.
Done when you can say why the index carries a version and who uses it.
Done when you can name work that must not run on the online path.