A recall-and-cost benchmark for an IVF index
2,000 random 4-dimensional points keep the run instant. Real embeddings will show different numbers, but the method (ground truth, sweep, pick by target) is the same one to use on them.
A faster index is only better if you know what it gave up. A trustworthy benchmark uses exact search as ground truth, real questions, a hardware-independent cost such as distances computed, and one knob turned at a time, then reads the curve for the cheapest setting that meets a recall target.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write cheapest_meeting(results, target) where results is a list of (setting, recall, cost) triples. Return the setting with the lowest cost among those with recall at or above target, or None if none qualifies. Print the answers for the results [(1, 0.800, 142), (2, 0.964, 265), (3, 0.984, 396)] with targets 0.95 and 0.99, and for an empty list.
For 50 test questions with k = 5, and for 1,000 questions with k = 10, print how many neighbours the benchmark checks and how much one miss moves the recall (4 decimals).
Explain it without notes
What are the parts of a trustworthy benchmark for an approximate index?
Why count distance computations as well as timing?
Looking at the nprobe sweep, how would you choose a setting, and what changes if the target recall goes from 0.95 to 0.99?
Why should two indexes be compared at the same recall rather than at the same knob value?
Expected output
nlist = 16, turning nprobe (exact search computes 2000 distances)
nprobe 1: recall@5 0.800, 142 distances
nprobe 2: recall@5 0.964, 265 distances
nprobe 3: recall@5 0.984, 396 distances
nprobe 4: recall@5 0.996, 521 distances
nprobe 6: recall@5 1.000, 768 distances
nprobe 8: recall@5 1.000, 1015 distances
nprobe 16: recall@5 1.000, 2016 distances
smallest nprobe with recall@5 of at least 0.95: 2
nprobe = 2, turning nlist
nlist 8: recall@5 0.944, 519 distances
nlist 16: recall@5 0.964, 265 distances
nlist 32: recall@5 0.896, 160 distances