Command Palette

Search for a command to run...

Hectal
Phase 8Intermediate9 of 14 in RAG and Vector Search

Generation with Sources

Turn retrieved passages into an answer you can trust: prompt layout, a context budget, checked citations, a calibrated refusal and conflicting sources.

Retrieval finds the passages. This phase is about the last step: giving them to a model so that the answer is grounded (it says only what the sources say), cited (every claim points at a source the reader can open), and honest (when the sources do not contain the answer, it says so instead of guessing).

You will lay out a prompt with numbered, delimited sources, count tokens against a budget, order the passages, and verify citations in code instead of trusting the model. You will replace the crude score bar from Phase 0 with a threshold measured on real scores from a real embedding model, handle two sources that disagree, ask for a validated JSON answer, and check an answer against its sources by hand.

LLM Engineering Phase 8 gave the short version of this subject (numbered sources, a citation check, a refusal). This phase goes deeper and does not repeat it. Everything ends in the AskDocs build: a budgeted prompt, checked citations, a refusal before the model is even called, and conflicts shown instead of hidden.

0/9 · 0%
9 topics ~3 h 17 code blocks & diagrams
Start with the first topic
1
8.1

Laying Out the Prompt

A RAG prompt has four jobs to do in one string: say what to do, hold the sources, mark where each one starts and ends, and ask the question. A fixed layout with numbered, delimited sources makes answers citeable and failures debuggable.

20 min 1 code practice

2
8.2

The Context Budget

Everything in the prompt competes for one fixed window, so sources get a token budget. You count tokens with a stated rough rule, pack passages best-first, decide what to do with one that does not fit, and order the rest. More context is not better.

21 min 2 code practice

3
8.3

Citations You Can Check

A citation is a claim about the evidence, and claims can be wrong. Parse the markers in the answer, check each against the numbers in the prompt, report unused and invalid ones, and show the reader the real source behind every marker.

20 min 2 code practice

4
8.4

Refusing Without Evidence

When the handbook does not contain the answer, the right output is 'I could not find that'. A score threshold measured on real answerable and unanswerable questions stops the call before the model is invoked, and replaces the crude word-count bar from Phase 0.

28 min 4 code practice

5
8.5

When Sources Disagree

Real handbooks hold old and new versions of the same page, and near-duplicate passages score almost the same. Dates, status and authority must travel with each passage into the prompt, and a genuine conflict is shown, not hidden.

21 min 2 code practice

6
8.6

Answer Formats

Prose is not always the right output. Quotes, steps and a validated JSON object with an answer, citations and a confidence let a product show sources and let code check the model. Streaming needs one extra care: hold back half-finished citation markers.

21 min 2 code practice

7
8.7

Checking the Answer Against the Sources

A valid citation does not prove a sentence is supported. Cheap checks in code (content-word overlap, numbers that must appear in the source) catch many unsupported claims, and a judge model gives a second opinion on the sentences that are flagged.

19 min 1 code practice

8
8.8

Build: Answers with Sources for AskDocs

AskDocs version 8 adds the whole generation side: a refusal gate on a measured threshold, a token-budgeted prompt with numbered sources and dates, a citation check, and a visible warning when a source is out of date.

22 min 1 code practice

9
8.9

Playbook: Grounded Answer Recipes

Five small, reusable pieces of the answer path: a prompt renderer, a context packer, a refusal gate, a citation verifier and a number checker, plus the one place a real chat call plugs in.

20 min 2 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Pack sources into a token budget

estimate_tokens(text) is given. Write pack(hits, budget, overhead=8). hits is a list of (doc_id, text) pairs, best first.

Beginner
2

Verify citations against the retrieved sources

Write check_citations(answer, n_sources) that returns three sorted lists of ints: (cited, invalid, unused).

Beginner
3

Refuse before the model is called

build_prompt(question, hits) is given. hits is a list of (doc_id, score, text) triples, best first. model is a function that takes the prompt and returns a string.

Intermediate
4

Validate a JSON answer

Write validate(raw, n_sources) that returns a list of problem strings (empty means valid). raw is the model's reply as text.

Intermediate
5

Flag sentences the source does not support

words(text) and STOP are given. Write unsupported(sentences, source, cutoff=70) returning the list of sentences that should be flagged, in the original order.

Advanced