Choosing a threshold from measured scores
The scores are the best cosine for each question, measured with all-MiniLM-L6-v2 in the search lab. Eleven questions are a demonstration, not evidence.
When the handbook does not contain the answer, the right output is 'I could not find that'. A score threshold measured on real answerable and unanswerable questions stops the call before the model is invoked, and replaces the crude word-count bar from Phase 0.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Using the scores ANSWERABLE = [47, 53, 75, 69, 43] and UNANSWERABLE = [38, 37, 33, 28, 15, 12], write best_threshold(answerable, unanswerable) that returns the threshold between 10 and 60 (step 5) with the fewest total mistakes (false refusals plus false answers), the lowest such threshold if tied. Print it.
Write decide(best_score, low, high) returning 'refuse' below low, 'confirm' from low up to but not including high, and 'answer' from high. Print the decision for scores 0.30, 0.42 and 0.60 with low=0.40 and high=0.50.
Explain it without notes
What are the two layers of refusing without evidence, and why have both?
How would you choose a score threshold, and what can go wrong?
Why can't you gate on a reciprocal-rank-fusion score?
A question about 'scooter riders' and helmets scores high. What does that tell you?
Expected output
lowest answerable: 0.43, highest unanswerable: 0.38
threshold 0.10: wrongly refused 0, wrongly answered 6
threshold 0.15: wrongly refused 0, wrongly answered 5
threshold 0.20: wrongly refused 0, wrongly answered 4
threshold 0.25: wrongly refused 0, wrongly answered 4
threshold 0.30: wrongly refused 0, wrongly answered 3
threshold 0.35: wrongly refused 0, wrongly answered 2
threshold 0.40: wrongly refused 0, wrongly answered 0 <- no mistakes
threshold 0.45: wrongly refused 1, wrongly answered 0
threshold 0.50: wrongly refused 2, wrongly answered 0
threshold 0.55: wrongly refused 3, wrongly answered 0
threshold 0.60: wrongly refused 3, wrongly answered 0