Topic 11.10
Build: AskDocs as a Service
In one line
AskDocs becomes a service: an ingestion path and a query path with an answer cache that is invalidated by document, a time budget that skips re-ranking when search is slow, a trace line for every request and a health check with a canary question.
Think of it like this
A restaurant has been cooking well in a test kitchen. Opening day needs more than recipes: a pass-through between kitchen and tables, a ticket for every order, a timer so no dish waits too long, and a morning check that the ovens are on. This build adds that around the pipeline you already have.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Service
- A program that stays running and answers requests, as opposed to a script that runs once.
- Health check
- A cheap test the platform runs to decide whether an instance should receive traffic.
- Canary question
- A fixed question with a known answer, used to check the whole path works.
- Budget
- The total time a request may take. Stages past the budget are skipped.
- Trace line
- One log line summarising the stages and timings of a request.
- Degrade gracefully
- Do a little less (skip a stage) under pressure instead of failing.
Step by step
01Step 1: separate ingest from ask
ingest(doc_id, text) is the only function that writes. ask(question) only reads the index and writes the cache. This is Topic 11.1's split at toy size.
02Step 2: make ingestion invalidate the cache
Each cached answer records the document it used. ingest pops that record and deletes the answers. The edit of D8 prints dropped 1 cached answer(s).
dropped = DEPS.pop(doc_id, set())
for key in dropped:
CACHE.pop(key, None)03Step 3: add the budget
Before re-ranking, check whether elapsed + RERANK_MS + GENERATE_MS still fits in BUDGET_MS. If it does not, skip the re-ranker and note it in the trace. Always generate: an answer without it is no answer.
04Step 4: return a trace line with every answer
trace: cache=miss search=20 rerank=80 generate=300 total=400. The cache hit line says cache=hit total=1. The same string would go to your logs with the request id.
05Step 5: add a health check
health() returns a failure if the index is empty or the canary question does not find D8. Call it from the platform's readiness probe so an instance with a broken index never receives traffic.
06Step 6: run it and read the story
The output reads like an incident-free day in order: fail, ingest, pass, miss, hit, edit, miss with the new price, slow search with a skipped stage, a refusal. Compare each line with the Step 3 and 4 rules.
Try it yourself
- 1
Tighten the budget
Set
BUDGET_MSto 400. Does the first price question still run the re-ranker? (20 + 80 + 300 is exactly 400: is that inside the budget?) Then set it to 399 and compare the trace. - 2
Remove the invalidation
Make
ingeststop dropping cached answers. Which line of the output proves the cache served a wrong price? - 3
Break the canary
Edit the canary question to one that D8 does not answer, and run. What does
health()say, and what would the platform do with an instance like that?
Code & diagrams
Retrieval is the keyword toy used since Phase 0 and the times are simulated. Replace the search and the generate step and the surrounding service code stays the same.
Expected output
health: FAIL: the index is empty
health: ok (index version 3, canary found D8)
trace: cache=miss search=20 rerank=80 generate=300 total=400 | The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
trace: cache=hit total=1 | The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
edit D8: dropped 1 cached answer(s)
trace: cache=miss search=20 rerank=80 generate=300 total=400 | The monthly pass costs 30 dollars and includes unlimited rides of up to 45 minutes. [D8]
trace: cache=miss search=150 rerank=skipped generate=300 total=450 | Shops open at 7 am and close at 9 pm every day [D2]
trace: cache=miss search=20 total=20 | I can't find that in the handbook.
requests=5 cache_hits=1 skipped_rerank=1Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Edit a document but keep its cached answers
In the Python tab, in ingest, replace the line dropped = DEPS.pop(doc_id, set()) # drop every cached answer that used this document with dropped = set() and run.
Myth vs fact
Myth
Production readiness is mostly about retrieval quality.
Fact
Quality is necessary. Staleness, latency, cost, observability and failure handling decide whether users can rely on it.
Myth
A health check should run a real question through the full model.
Fact
It should be cheap and frequent. Check the index and a retrieval canary; run a full-answer canary less often, because it costs money.
Myth
Skipping a stage under load is cheating.
Fact
It is a designed degradation, logged in the trace, that keeps the response within the budget. Silently skipping would be cheating; recording it is engineering.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep readiness (can I serve?) and liveness (am I stuck?) probes separate (see the Docker course,
/docker, and Kubernetes,/kubernetes). A slow index load should make an instance not ready, not trigger a restart loop. - ▸
Return the trace line to the client in a header or field only in internal builds. Externally, keep it in logs.
- ▸
Use single-flight on cache misses, so fifty copies of a popular new question trigger one generation (Topic 11.3).
- ▸
The budget should also cancel work that is no longer useful: if the client disconnected, stop (LLM Engineering Topic 5.5).
Remember this
- 1
Why the product needs this now. Phases 0 to 10 made AskDocs find, rank and answer well and safely. None of that helps if it is slow under load, serves a price from yesterday, or breaks silently after a deploy. This step adds the production parts of this phase around the same
retrieveandanswerinterface. Before it, AskDocs was a pipeline you ran from a script: it answered well, but nothing kept it current, remembered an answer, watched its own speed or told you when it was broken. After it, AskDocs can take document edits while it runs, answer repeat questions from a cache that never serves an edited document's old answer, stay inside a time limit, explain every request in one line, and say whether it is healthy. - 2
What the step adds. (1) An ingestion path (
ingest) that stores the document, rebuilds its index entry, bumps the index version and drops cached answers that used it (Topics 11.1 and 11.3). (2) A query path (ask) that checks the answer cache first. (3) A time budget of 500 ms: a request that has spent too long searching skips the re-ranker (Topic 11.4). (4) A trace line per request with the time of each stage (Topic 11.6). (5) A health check with a canary question (Topic 11.9). - 3
How it plugs in. The keyword retrieval is the same toy as before behind
retrieve/search. The cache wraps the wholeask. The budget wraps the two later stages. Nothing about the retrieval algorithm changes, which is the point: the service parts are independent of how good retrieval is. - 4
Reading the output. At the start
healthfails (empty index). After ingesting three documents it passes with index version 3. The first price question misses the cache and takes 400 ms: search 20, re-rank 80, generate 300. The same question in different capitals hits (1 ms). After D8 is edited, one cached answer is dropped, and the next ask returns the new price (30 dollars) instead of the old one. A question asked while search takes 150 ms skips the re-ranker (150 + 80 + 300 would be 530 ms, over the 500 ms budget) and finishes in 450. The scooter question is refused, and the refusal is not cached. - 5
What is still missing. Real clocks and real model calls (the times here are simulated), a persistent queue and workers for ingestion, authentication and per-tenant filters (Phase 10 added the guards, wire them into
search), a shared cache like Redis instead of a dictionary, and the evaluation harness (Phase 9) running in CI before each deploy. Phase 12 adds smarter retrieval patterns and Phase 13 assembles everything. - 6
The lesson of this build: every production part sits at the edge of the pipeline, not inside it. That keeps the core testable on its own and lets you add or remove a part (the cache, the budget) with a switch.
Explain it without notes
What does AskDocs-as-a-service add around retrieval, and why at the edges?
How does the example avoid serving a stale price after a document edit?
What does the time budget do, and what does it never skip?
What should a health check test for a RAG service?
Practice
Write stages_run(search_ms, budget_ms, rerank_ms=80, generate_ms=300) returning the list of stages that run (search always, rerank only if search plus rerank plus generate fit the budget, generate always). Print it for search times 20 and 150 with a budget of 500.
Write hit_rate(hits, requests) returning a percentage as a float, and 0.0 when there were no requests. Print it with one decimal for (1, 5) and (0, 0).
Trade-offs
- ↔
An answer cache: faster and cheaper, with staleness risk managed by invalidation.
- ↔
A hard time budget: predictable latency, with some answers made without the re-ranker.
- ↔
A cheap canary: frequent and low cost, and it cannot prove the model's answer is good.
Done when you can
Done when you can run the service example in all three languages and explain every line of its output.
Done when you can show why an edit drops exactly the cached answers that used the document.
Done when you can explain how the budget decides whether to re-rank.
Done when you can say what the health check tests and why it fails on an empty index.
Done when you can name what the toy service still lacks for real production.