The refusal gate in answer()
RETRIEVED stands in for retrieve(): the scores are measured values, so the example prints what the real model's scores would decide.
When the handbook does not contain the answer, the right output is 'I could not find that'. A score threshold measured on real answerable and unanswerable questions stops the call before the model is invoked, and replaces the crude word-count bar from Phase 0.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Using the scores ANSWERABLE = [47, 53, 75, 69, 43] and UNANSWERABLE = [38, 37, 33, 28, 15, 12], write best_threshold(answerable, unanswerable) that returns the threshold between 10 and 60 (step 5) with the fewest total mistakes (false refusals plus false answers), the lowest such threshold if tied. Print it.
Write decide(best_score, low, high) returning 'refuse' below low, 'confirm' from low up to but not including high, and 'answer' from high. Print the decision for scores 0.30, 0.42 and 0.60 with low=0.40 and high=0.50.
Explain it without notes
What are the two layers of refusing without evidence, and why have both?
How would you choose a score threshold, and what can go wrong?
Why can't you gate on a reciprocal-rank-fusion score?
A question about 'scooter riders' and helmets scores high. What does that tell you?
Expected output
Q: I forgot my login, how do I get back in?
best score 0.47, sources used: 1
A: To reset your password, open the app, tap Forgot password and follow the link we email you. [1]
Q: Can I get my money back if I cancel?
best score 0.53, sources used: 1
A: A ride cancelled at least 2 hours before its start time is refunded in full. [1]
Q: Do you rent scooters?
best score 0.38, sources used: 0
A: I could not find that in the handbook.
Q: Can I bring my dog on the bike?
best score 0.37, sources used: 0
A: I could not find that in the handbook.
model calls: 2 of 4 questions