Command Palette

Search for a command to run...

Hectal
Phase 11Advanced12 of 14 in RAG and Vector Search

RAG in Production

From a notebook that works to a service that keeps working: two paths, freshness, caches, budgets, cost, traces, scaling, model changes and incident drills.

A RAG system that answers well on your laptop is not yet a service. Real users ask at the same moment, documents change while they ask, a provider is slow on a Tuesday, a bill arrives, and someone asks "why did it say that?" three weeks later. This phase is about everything around retrieval and generation that decides whether the system keeps working.

You will split the system into its ingestion path and its query path, measure how stale an answer may be, cache every stage without serving wrong answers, give each stage a latency budget, work out the cost of one question by hand, log and trace what retrieval did, scale the index with shards and replicas, change the embedding model without an outage, and rehearse four incidents with runbooks. Every example is a simulation with a counter for the clock and example prices, so it runs identically in three languages.

The phase ends by turning AskDocs into a service: the two paths wired together with an answer cache, a time budget, traces and a health check, and a playbook of recipes you can copy. The Observability course (/observability) covers the general tools for logs, metrics and traces; System Design (/system-design) covers caching, sharding and replication in general. Here they are applied to retrieval.

0/11 · 0%
11 topics ~4 h 15 code blocks & diagrams
Start with the first topic
1
11.1

Architecture of a RAG Service

A RAG service is two programs that share data: an ingestion path that turns documents into an index when they change, and a query path that answers a question in a few seconds. Knowing which parts are online, which are batch and where state lives is the first production decision.

21 min 1 code practice

2
11.2

Freshness

An index is a photograph of the documents at some moment. Freshness is how old that photograph may be when a question arrives. You decide the allowance, measure the lag between an edit and its appearance in answers, and show the reader the "as of" date.

20 min 1 code practice

3
11.3

Caching Every Stage

Most questions are repeats, so each stage of the pipeline (embedding, retrieval, the final answer) can remember its result. The hard part is the key: it must contain everything that changes the result, so a changed document, model or prompt never serves an old answer.

22 min 2 code practice

4
11.4

Latency Budgets

A budget gives each stage of the query path a time limit, the way a trip has a time for each leg. With budgets you can see which stage is spending too much, decide what to cut first, and make time to first token feel fast even when the whole answer takes seconds.

20 min 1 code practice

5
11.5

What RAG Costs

RAG has a one-off cost (embedding the documents), a small monthly cost (storing the vectors) and a per-question cost (embedding, re-ranking and generation). Working out the cost of one question by hand shows which lever saves the most.

21 min 1 code practice

6
11.6

Observing Retrieval

When an answer is wrong, you need to know what retrieval did: which passages came back, with what scores, and how long each stage took. Structured log lines and trace spans record that, and a few metrics tell you when to look.

24 min 2 code practice

7
11.7

Scaling

When the corpus or the traffic outgrows one machine, you split the index into shards (to hold more) and copy it into replicas (to answer more, and to survive failures). Every query then goes to every shard, and the results are merged.

23 min 1 code practice

8
11.8

Changing the Embedding Model

Vectors from different embedding models live in different spaces and cannot be compared. Changing the model means building a second index, filling it in the background, measuring it on your test questions, switching only if it is better, and keeping a way back.

27 min 2 code practice

9
11.9

Incident Drills

Four things go wrong in every RAG service sooner or later: stale answers, an empty index after a bad deploy, a leak report and a cost spike. For each, write a runbook now: how to notice, what to check first and what to do, in priority order.

21 min 1 code practice

10
11.10

Build: AskDocs as a Service

AskDocs becomes a service: an ingestion path and a query path with an answer cache that is invalidated by document, a time budget that skips re-ranking when search is slow, a trace line for every request and a health check with a canary question.

22 min 1 code practice

11
11.11

Playbook: Production RAG Recipes

Four recipes to copy into any RAG project: a traced request, cache keys with invalidation, a latency budget table, and a migration runbook for changing the model. Plus a launch checklist for the production side.

20 min 2 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Measure index lag

Write index_lag(edited_at, indexed_at) taking two dicts (document id to tick) and returning a dict of document id to lag (indexed_at - edited_at) for every document that appears in both dicts.

Beginner
2

A cache with invalidation by document

Complete the class AnswerCache. put(key, answer, doc_ids) stores the answer and remembers that each document in doc_ids was used. get(key) returns the answer or None.

Intermediate
3

Find the stage furthest over budget

Write percentile(values, pct) using the nearest-rank rule: sort the values and return the one at position ceil(pct * n / 100) (counting from 1). Use whole-number arithmetic: (pct * n + 99) // 100.

Intermediate
4

Merge results from shards

Write scatter_gather(shards, score, k). shards is a list of lists of document ids, and score(doc_id) returns a number. From each shard take its best k documents, then merge them and return the best k document ids overall.

Intermediate
5

Triage an incident snapshot

Write triage(m) taking a dict with the keys cross_tenant_hits, index_docs, docs_expected, oldest_wait and cost_micro, and returning the list of alert names that apply, in this priority order:

Advanced