Frozen knowledge, pasted documents and the size arithmetic
A fake model that answers from the document when it is given one, otherwise from a frozen guess. The sizes use a rule of thumb (about 3 words per 4 tokens) and an example price.
A model knows only its training text, which ended months ago and never included your private documents. RAG finds the relevant pages at question time and puts them in the prompt, which is cheaper, fresher and easier to check than retraining.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
A manual has 250 pages of 300 words. Using 3 words for every 4 tokens, how many tokens is it? What percent fits in a 16,000-token window? Compute in Python and print both (one decimal for the percent).
At an example price of 0.50 dollars per million input tokens, what does it cost to paste a 100,000-token manual into every one of 20,000 questions? Print the total to two decimals.
Explain it without notes
Why can a model not answer questions about a company's private documents, and what does RAG change?
Why is retrieval usually better than fine-tuning for teaching a model facts?
If a model has a very large context window, why would you still retrieve instead of pasting everything?
Expected output
question: How long do I have to return a bike?
without documents -> From memory: 14 days
with the shop document -> From the document: Unused bikes can be returned within 30 days with the receipt for a full refund.
manual size: 120000 words, about 160000 tokens
context window: 8000 tokens, so 5.0 percent of the manual fits
paste everything: 0.0800 dollars per question
retrieve 3 chunks of 150 words (600 tokens): 0.0003 dollars per question