PHASE 8Intermediate ~22 min· topic 8 of 9

Topic 8.8

Build: Answers with Sources for AskDocs

In one line

AskDocs version 8 adds the whole generation side: a refusal gate on a measured threshold, a token-budgeted prompt with numbered sources and dates, a citation check, and a visible warning when a source is out of date.

Think of it like this

A restaurant kitchen that already has good ingredients (retrieval) and now adds the pass, where each plate is checked against the order before it goes out. Wrong order, missing garnish, a dish nobody ordered: it goes back. The plate that reaches the customer has been checked.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Pipeline stage: One step of the answer path: retrieve, gate, pack, prompt, model, verify.
Gate: The check that returns a refusal before the model is called.
Packer: The step that keeps passages while they fit the token budget.
Verifier: Code that checks the model's output, here its citations.
Stand-in: A scripted fake with the same inputs and outputs as the real component.
Superseded source: A passage replaced by a newer version, kept and labelled in the prompt.

Step by step

01Step 1: retrieve with scores

retrieve(question, k) returns (doc_id, score) pairs, best first. The example's table holds the cosines measured in the lab.

02Step 2: gate on the measured threshold

Drop hits below 0.40. If nothing remains, print the gate line and return the refusal without calling the model.

03Step 3: pack the survivors into the budget

Charge each passage its estimated tokens plus the 8-token overhead and keep the ones that fit in 120 tokens. Two passages (58 tokens) fit for the pass question.

Step 3: pack the survivors into the budgetdiagram

Rendering diagram…

04Step 4: label the versions in the prompt

The prompt header shows effective 2024-03-01, superseded for the old price list and effective 2025-06-01, current for the new one.

05Step 5: call the model and verify its citations

The stand-in cites the first current source and names the superseded one. The verifier checks every marker is between 1 and the number of kept sources.

06Step 6: read the trace

For each question the output prints the hits with scores, the kept sources with tokens used, the citation verdict and the answer. If an answer is wrong you can see which stage to blame.

Try it yourself

1
Change the budget
Set BUDGET = 45 and run. Which passages does the pass question keep now? Is the answer still right, and what does that tell you about budgets and ordering?
2
Ask a new question
Add a question to SCORES with made-up scores for D6 (0.55) and D3 (0.20). Predict the output before you run it.
3
Make the verifier bite
Make fake_model return ... [5] for one question. What does the citations line say? Decide what answer should do then, and implement it.

Code & diagrams

AskDocs version 8: gate, budget, labelled sources, checked citations New tab

The scores in SCORES are real cosines from the search lab, rounded. The model is a scripted stand-in that follows the prompt's rules. Re-measure the threshold on the whole handbook before trusting 0.40.

Expected output

Q: I forgot my login, how do I get back in?
  hits: D3 0.47, D1 0.11, D6 0.11
  kept: D3 (39 of 120 tokens)
  citations: ok
  A: To reset your password, open the app, tap Forgot password and follow the link we email you. [1]
Q: What does the monthly pass cost?
  hits: D16 0.76, D8 0.75, D6 0.22
  kept: D16, D8 (58 of 120 tokens)
  citations: ok
  A: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [2] ([1] is an older version and no longer applies.)
Q: Can I get my money back if I cancel?
  hits: D1 0.53, D6 0.33, D3 0.23
  kept: D1 (39 of 120 tokens)
  citations: ok
  A: A ride cancelled at least 2 hours before its start time is refunded in full. [1]
Q: Do you rent scooters?
  hits: D1 0.27, D6 0.24, D16 0.23
  gate: best score 0.27 is below 0.40, model not called
  A: I could not find that in the handbook.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Lower the threshold

In the Python tab, change THRESHOLD = 0.40 to THRESHOLD = 0.20. (Last question of the output from the Python run.)

terminal

$ Run

── what you'll see ──

Q: Do you rent scooters?

hits: D1 0.27, D6 0.24, D16 0.23

kept: D1, D6, D16 (99 of 120 tokens)

citations: ok

A: A ride cancelled at least 2 hours before its start time is refunded in full. [1]

Myth vs fact

Myth

Once retrieval is good, generation is easy.

Fact

Generation has its own failures: unbudgeted prompts, missing refusals, bad citations and conflicts. Each has to be handled on purpose.

Myth

The product should always give an answer.

Fact

A refusal is a valid and valuable output. The scooter question is the product working.

Myth

Checks make the model unnecessary.

Fact

Checks decide whether to trust the model's output. They cannot write the answer.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Print the trace for every request in development and log it in production: hits with scores, kept ids, tokens used, citation verdict. Most generation bugs are found in the trace, not in the answer text.
▸
Put the stages behind small functions with the names used here (retrieve, pack, build_prompt, fake_model or the real call, verify). Each can then be unit tested alone and swapped as the course adds Phases 9 to 11.
▸
A version-8 fake model that always follows the rules proves the plumbing, not the model. Run the same four questions against a real model (the lab, or the playbook's call) and compare citations and refusals before you ship.

Remember this

1
Why the product needs this now. Until this phase AskDocs found passages and handed the first one to a stand-in model. Phases 1 to 7 made retrieval better. Nothing yet decides whether to answer at all, how many passages fit, how to cite them, or what to do when two disagree. Those are generation decisions, and each one has a failure you have seen in this phase.
2
What the step adds. Inside answer(question): (1) retrieve returns scored hits; (2) a gate drops hits below 0.40, the threshold from Topic 8.4, and refuses without calling the model if none remain; (3) a packer keeps hits that fit a 120-token budget (Topic 8.2); (4) build_prompt numbers the sources, adds each one's effective date and status (Topic 8.5) and puts the question last; (5) the model answers; (6) a verifier checks the citations (Topic 8.3).
3
The interface stays. retrieve(question, k) and answer(question) keep their shapes. build_prompt gains the metadata it needs, and the caller never notices. In the example retrieve is a stand-in with scores measured in the lab, so the output shows what the real model's numbers would decide; Phase 3 and 5's real index plugs in behind the same call.
4
The four questions. A paraphrase ('I forgot my login...') is answered from D3. The pass-price question retrieves both price lists, labels the 2024 one superseded and answers with the current price while pointing out the older source. A refund question is answered from D1. The scooter question is refused by the gate and the model is never called.
5
What is still missing. The model is a scripted stand-in (a real one plugs in through the call shown in the playbook). Grounding checks from Topic 8.7 are not wired in. There is no evaluation set (Phase 9), no access control (Phase 10) and no caching or latency budget (Phase 11). The corpus has only five passages, so the threshold is only roughly right: re-measure it on the full handbook.

Explain it without notes

Walk through the stages of AskDocs version 8 for the pass-price question.

Why is the scooter question refused before the model is called?

What does the verifier check, and what does it not?

Which parts of the answer path are real in the example and which are stand-ins?

Practice

Add a question "Is the handbook available in French?" to the example's SCORES with D1 at 0.12 and D3 at 0.10. Predict the output of answer, then run it.

Write explain_refusal(hits, threshold) returning the sentence best score X.XX is below Y.YY for a list of (doc_id, score) pairs, best first. Print it for [("D1", 0.27), ("D6", 0.24)] and 0.40.

Trade-offs

↔
Gating before the call: cheaper and predictable, but depends on a threshold you must keep current.
↔
A tight budget: faster and cheaper, but a relevant passage can be cut. A loose one costs more and adds noise.
↔
Labelling old versions instead of dropping them: supports history questions, at the cost of tokens and relying on the model to follow the rule.

Done when you can

Done when you can run the example and explain each line of the trace.
Done when you can name the interface each stage sits behind and what changed in it.
Done when you can explain why the scooter question never reaches the model.
Done when you can list what version 8 still lacks and which phase adds it.

Command Palette

Words you'll meet

Step by step

01Step 1: retrieve with scores

02Step 2: gate on the measured threshold

03Step 3: pack the survivors into the budget

04Step 4: label the versions in the prompt

05Step 5: call the model and verify its citations

06Step 6: read the trace

Try it yourself

Change the budget

Ask a new question

Make the verifier bite

Code & diagrams

Break it on purpose

Lower the threshold

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can