AskDocs v3: vectors, two indexes, a similarity bar, a recall check
The vectors are a toy with four named directions, and the embedder returns prepared values for known texts. Real numbers from a real model are in the search labs of this phase and of Phase 2.
AskDocs version 3: an Embedder turns passages and questions into vectors, an exact index and an IVF index sit behind the same add and search interface, retrieval keeps only hits above a similarity bar, and recall against the exact index is measured. The login question is finally answered correctly.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write recall_against_exact(exact, other, k) where both arguments are lists of ranked id lists (one per query). Return the average, over queries, of the fraction of the exact top k that other also has in its top k. Print recall@1, @2 and @3 for exact = [["D1", "D8", "D6"], ["D3", "D2", "D9"]] and other = [["D1", "D8", "D5"], ["D3", "D2", "D9"]].
Write evidence(hits, min_sim) where hits is a list of (doc_id, similarity) pairs. Return the hits at or above min_sim, or the refusal text "I can't find that in the handbook." when none qualify. Print the result for three calls: two good hits, one weak hit (0.61), and no hits, with min_sim = 0.90.
Explain it without notes
Why do FlatIndex and IvfIndex share the same two methods, and what does that buy?
In this build, why does the login question work now when version 0 failed it?
Why does the scooters question get the refusal, and what would change if MIN_SIM were 0?
What does the recall check measure, and why is it weak with only twelve vectors?
Expected output
Q: Can I get my money back if I cancel?
exact: D1 (0.991), D8 (0.974)
ivf: D1 (0.991), D8 (0.974)
answer: A ride cancelled at least 2 hours before its start time is refunded in full. [D1]
Q: What does the monthly pass cost?
exact: D8 (0.991), D1 (0.963)
ivf: D8 (0.991), D1 (0.963)
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Q: I forgot my login, how do I get back in?
exact: D3 (0.988)
ivf: D3 (0.988)
answer: To reset your password, open the app, tap Forgot password and follow the link we email you. [D3]
Q: Do you rent scooters?
exact: none
ivf: none
answer: I can't find that in the handbook.
recall@3 of the IVF index against the exact index, using each passage as the query:
nprobe 1: 0.917
nprobe 2: 1.000
nprobe 3: 1.000