Topic 8.4
Refusing Without Evidence
In one line
When the handbook does not contain the answer, the right output is 'I could not find that'. A score threshold measured on real answerable and unanswerable questions stops the call before the model is invoked, and replaces the crude word-count bar from Phase 0.
Think of it like this
Two librarians. Ask the first for a book they do not have and they say 'we do not have that'. The second hands you whatever is nearest on the shelf, with a confident smile. The second librarian is friendlier and far more dangerous. A RAG system without a refusal is the second librarian.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Refusal
- A deliberate 'I could not find that' answer when the sources do not contain the answer.
- Threshold
- The lowest score at which a passage still counts as evidence.
- Gate
- A check that runs before an expensive step and can stop the request early.
- Answerable
- A question the documents actually contain the answer to.
- False refusal
- Refusing a question the handbook can answer.
- False answer
- Answering a question the handbook cannot answer.
- Calibration
- Choosing a threshold from measured examples so that it separates the cases that matter.
Step by step
01Step 1: collect two lists of questions
Write questions the handbook answers (with their paraphrases) and questions it does not: other products, other topics, near-misses. Phase 9 turns this into a golden set.
02Step 2: measure the best score for each
Run each question through the retriever and write down the highest cosine. The search lab below does this for you with the real model: type a question and read the top Cosine.
03Step 3: sweep thresholds and count mistakes
For each candidate threshold, count answerable questions that would be refused (false refusals) and unanswerable ones that would be answered (false answers). The first example prints the table.
04Step 4: pick a threshold and a margin
Choose a value inside the gap, closer to the side whose mistakes cost more. Write down the model name and date with it.
05Step 5: put the gate in answer()
The second example runs four questions through the gate. Two are answered and two are refused, and the model is called only twice.
06Step 6: try to break it
Use the first lab to write a question the handbook cannot fully answer that still scores high. That is a question your gate will let through, and the reason the prompt also tells the model to refuse.
Try it yourself
- 1
Find the gap yourself
In the first lab, type five questions the handbook cannot answer and five it can. Note each best Cosine. Is there a threshold with no mistakes in your sample? How wide is the gap?
- 2
Move the bar
In the gate example, change
THRESHOLDto 0.50 and run. Which question that used to be answered is now refused? Which kind of mistake did you trade for which? - 3
Take away the exact sentence
In the second lab, replace the instruction with 'Answer using the sources.' and send. Does the model still refuse? Why does the course insist on an exact sentence?
Code & diagrams
The scores are the best cosine for each question, measured with all-MiniLM-L6-v2 in the search lab. Eleven questions are a demonstration, not evidence.
Expected output
lowest answerable: 0.43, highest unanswerable: 0.38
threshold 0.10: wrongly refused 0, wrongly answered 6
threshold 0.15: wrongly refused 0, wrongly answered 5
threshold 0.20: wrongly refused 0, wrongly answered 4
threshold 0.25: wrongly refused 0, wrongly answered 4
threshold 0.30: wrongly refused 0, wrongly answered 3
threshold 0.35: wrongly refused 0, wrongly answered 2
threshold 0.40: wrongly refused 0, wrongly answered 0 <- no mistakes
threshold 0.45: wrongly refused 1, wrongly answered 0
threshold 0.50: wrongly refused 2, wrongly answered 0
threshold 0.55: wrongly refused 3, wrongly answered 0
threshold 0.60: wrongly refused 3, wrongly answered 0RETRIEVED stands in for retrieve(): the scores are measured values, so the example prints what the real model's scores would decide.
Expected output
Q: I forgot my login, how do I get back in?
best score 0.47, sources used: 1
A: To reset your password, open the app, tap Forgot password and follow the link we email you. [1]
Q: Can I get my money back if I cancel?
best score 0.53, sources used: 1
A: A ride cancelled at least 2 hours before its start time is refunded in full. [1]
Q: Do you rent scooters?
best score 0.38, sources used: 0
A: I could not find that in the handbook.
Q: Can I bring my dog on the bike?
best score 0.37, sources used: 0
A: I could not find that in the handbook.
model calls: 2 of 4 questionsTry: Press Search for 'Do you rent scooters?' and read the best Cosine. Then try your own unanswerable questions and note their best scores. Finally solve the goal: write a question the handbook cannot fully answer that still scores at least 0.45 against the helmet passage. That is a false answer a threshold of 0.40 would let through. Switch Rank by to Hybrid and compare the Fused column across your questions.
Goals
- A question the handbook cannot fully answer scores at least 0.45 against the helmet passage
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Try: Press Search, then press Answer. The model gets the top passages and the question, with an instruction to reply with a fixed sentence when the sources do not contain the answer. Does it? Then edit the instruction: remove the fixed sentence and see what the model does instead. Change the question to 'Can I get a refund?' and send again.
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Answer from the sources · real model
Search first, then sign in (free) to send the retrieved passages to a real model.
- Refuses with the fixed sentence
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A bar that is too low
In the Python tab of the gate example, change THRESHOLD = 0.40 to THRESHOLD = 0.30. (Tail of the output from the Python run.)
Myth vs fact
Myth
A cosine of 0.5 means 50 percent sure.
Fact
Cosine is a similarity, not a probability. The same number means different things for different questions and models.
Myth
The instruction 'say you do not know' is enough.
Fact
Models often answer anyway. The gate before the call is free and deterministic, and the instruction covers what the gate lets through.
Myth
One threshold works forever.
Fact
It is tied to the embedder, the chunking and the language. Re-measure when any of them change.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Eleven questions cannot support a threshold. One added question can move the gap, and the real gap may close entirely (the scooter and puncture scores above are only 0.04 apart). Re-measure on your own data and model with at least a few dozen answerable and a few dozen unanswerable questions, keep some aside to test the chosen value, and expect to pick a bar that trades some false answers for fewer false refusals rather than a clean zero.
- ▸
Refusals are data. Log every refusal with its question and top scores. A cluster of refused questions that people keep asking is a list of documents you should write (Topic 9.8).
- ▸
A cheap user-friendly refusal says what was searched and what to do next ('Try asking about refunds, or contact support'), rather than just the fixed sentence. Keep the exact sentence for machines (the check in Topic 8.3) and render a friendlier message from it.
- ▸
Per-topic thresholds are possible when the documents differ a lot (short FAQ answers score higher than long policy pages). Do it only if the golden set shows a gain; one global threshold is easier to explain and monitor.
Remember this
- 1
Why refuse. Given weak or irrelevant sources, a model will often still produce a smooth, confident answer. A wrong answer that sounds sure costs more than 'I could not find that': the rider gets the wrong refund rule, the support team gets an angry email. So the product needs a way to say no, and it needs it to be deliberate, testable and cheap.
- 2
Two layers. The first is a gate before the model: if the best retrieved score is below a threshold, do not call the model at all, and return the fixed refusal sentence. It is deterministic, free and instant. The second is the instruction: the prompt tells the model to reply with the exact sentence when the sources it was given do not contain the answer. The gate catches questions with no related passages. The instruction catches the ones that slipped past it.
- 3
Replacing Phase 0's crude bar. In Phase 0 a passage counted as evidence if it shared two words with the question (
MIN_SCORE = 2). That number meant nothing across questions: long questions share more words, and paraphrases share none. Now that you have embeddings, the score is a cosine similarity, which is comparable across questions. The threshold must still be measured, not guessed. - 4
A score is not a probability. Recall from Topic 2.2: cosine similarity is a number near 1 for texts with the same meaning and near 0 for unrelated ones. In the lab with the real embedding model, 'I forgot my login, how do I get back in?' scores about 0.47 against the password passage, which is exactly the right passage. 'Do you rent scooters?' scores about 0.38 against the helmet passage, which is *not* an answer. 'What happens if my bike gets a puncture?' scores only about 0.43 against its best passage, and that one is the damage passage, not the flat-tyre passage (the right one scores about 0.39, under the 0.40 bar the example ends up with, so even a tuned gate can refuse a question the handbook answers). A cosine of 0.5 means different things for different questions and different models.
- 5
How to measure a threshold. Collect questions the handbook can answer and questions it cannot. Record the best score for each. The example below does this for 5 answerable and 6 unanswerable questions, using scores measured in the search lab. The lowest answerable score is about 0.43, the highest unanswerable about 0.38. Sweeping thresholds shows exactly one setting in this tiny sample with no mistakes: 0.40, inside a gap only 0.05 wide (0.38 to 0.43). Eleven questions are far too few to trust (Phase 9 builds a proper golden set), but the method is the real one. The 0.40 is this small sample's number, not a rule. It comes from one model (all-MiniLM-L6-v2) and 11 questions about 12 handbook passages. On your own documents, questions and embedding model, measure it again; do not copy 0.40.
- 6
The threshold belongs to the model. Another embedding model, or the same model with titles added to the text, has different scores. Re-measure whenever the embedder, the chunking or the language changes. Store the threshold with the name of the model it was measured on.
- 7
Gate on cosine, not on fused scores. Reciprocal rank fusion (Topic 6.3) scores depend on *rank*, not similarity. In the lab with the hybrid ranker, the top fused score for the scooter question is about 0.0325 and for the pass-price question about 0.0328: nearly the same, although one question has an answer and the other has none. Gate on the cosine score, or on a calibrated re-ranker score.
- 8
Wrong answer or no answer? The threshold is a cost decision. For refund rules, a wrong answer is expensive: set the bar higher and accept more refusals. For a casual 'where is the shop' bot, a refusal annoys more than a slightly loose answer. Some teams use two bars: below the low bar, refuse; between the bars, answer but ask the user to confirm; above the high bar, answer.
- 9
A score cannot see what is missing. 'Are helmets free for scooter riders?' scores about 0.72 against the helmet passage because the topic matches, yet the handbook says nothing about scooters. The gate lets it through, and the model must notice. This is why the instruction exists and why Topic 8.7 checks the answer against its sources.
Explain it without notes
What are the two layers of refusing without evidence, and why have both?
How would you choose a score threshold, and what can go wrong?
Why can't you gate on a reciprocal-rank-fusion score?
A question about 'scooter riders' and helmets scores high. What does that tell you?
Practice
Using the scores ANSWERABLE = [47, 53, 75, 69, 43] and UNANSWERABLE = [38, 37, 33, 28, 15, 12], write best_threshold(answerable, unanswerable) that returns the threshold between 10 and 60 (step 5) with the fewest total mistakes (false refusals plus false answers), the lowest such threshold if tied. Print it.
Write decide(best_score, low, high) returning 'refuse' below low, 'confirm' from low up to but not including high, and 'answer' from high. Print the decision for scores 0.30, 0.42 and 0.60 with low=0.40 and high=0.50.
Trade-offs
- ↔
High threshold: fewer wrong answers, more false refusals and support tickets. Low threshold: friendlier, but confident wrong answers.
- ↔
Gate before the model: saves cost and is predictable, but it only sees scores. The instruction sees content but costs a model call and can be ignored.
- ↔
One global threshold: simple to monitor. Per-topic thresholds: fit the data better, but need more examples and more upkeep.
Done when you can
Done when you can explain why a score is not a probability, with an example.
Done when you can measure answerable and unanswerable scores and sweep a threshold.
Done when you can put a gate before the model call and a fixed refusal sentence in the prompt.
Done when you can say why the threshold must be re-measured when the embedder changes.