PHASE 7Intermediate ~23 min· topic 1 of 9

Topic 7.1

The Question Is the Weak Link

In one line

The index was built carefully, but the question is whatever the user typed: short, vague, misspelt, or pointing back at something said earlier. A retriever can only compare the text it is given, so a weak question beats a strong index.

Think of it like this

You walk into a library and say "that book about the thing, the blue one". The librarian may be excellent and the shelves perfectly sorted, but you gave nothing to search with. Ask "a beginner's book about beekeeping" and the same librarian finds it in a minute. The shelves did not change. Only the question did.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Query: The text sent to the retriever. It can differ from what the user typed once a query layer has rewritten it.
Query layer: Code between the user's text and the retriever that cleans, rewrites, splits, routes or filters the question.
Content word: A word that carries meaning for search, as opposed to stop words like "the" and "do".
Pointer word: A word like "it", "that" or "one" whose meaning is somewhere else in the conversation.
Vocabulary mismatch: The user and the document say the same thing in different words.
Compound question: One message that asks two or more things.
Silent failure: A wrong result with no error: the system answers anyway.

Step by step

01Step 1: the index is fine, the question is not

Take the handbook as it is. Ask the single word "refund". The refunds passage says "refunded", so a word-matching retriever finds nothing. The index did nothing wrong. It was handed one short word in a different form.

02Step 2: a question can retrieve the wrong passage

"hours" matches "2 hours" in the refunds passage, not the opening-hours passage (which says "open" and "close"). A retrieved passage with a score is not proof the question was understood.

03Step 3: flag the weak shapes with rules

Three rules catch most weak questions: very few content words, a pointer word, a content word that appears in no passage. They never decide anything on their own. They only say "look at this one".

problems.pywhole filepython

def problems(question):
    found = []
    if len(content_words(question)) <= 1:
        found.append("short")
    if POINTERS & set(tokenize(question)):
        found.append("points back")
    unknown = sorted(content_words(question) - VOCAB)
    if unknown:
        found.append("unknown: " + " ".join(unknown))
    return found or ["ok"]

04Step 4: read the table

Run the example. The monthly pass question is fine. "hours" is short and lands on the wrong passage. "refund" is short and has an unknown word. "pasword rest" is all unknown words. The long money-back question has three unknown words even though it is a perfectly good question: it uses different words from the handbook. "what about it?" has nothing to search with.

05Step 5: where the layer sits

Each weak shape has a matching tool in this phase. Nothing after the layer changes.

Step 5: where the layer sitsdiagram

Rendering diagram…

06Step 6: decide with numbers

Count how many of your real questions get a flag. If it is 2 in 100, a query layer is a distraction. If it is 30 in 100, it is probably the best next investment. Topic 9.2 shows how to collect the questions.

Try it yourself

1
Predict the hit
Before running, predict the best hit for "Do you rent scooters?" and for "helmet". Add them to the list and compare. Which of the two is a question the handbook cannot answer, and does the retriever notice?
2
Add your own weak questions
Add three questions you might type in a hurry (a typo, one word, a pronoun). Which flag fires? Which weak question gets no flag at all but still retrieves the wrong passage?
3
Open the table of what is unknown
For the long money-back question, the unknown words are back, cancel and money. Add refunded to the question and run again. Which flag disappears, and does the best hit change?

Code & diagrams

Spot weak questions with rules New tab

The retriever here counts shared content words over all twelve handbook passages (stop words ignored). The point is the flags column, not the retriever.

Expected output

question                              best hit  problems
What does the monthly pass cost?      D8 (2)    ok
hours                                 D1 (1)    short
refund                                none      short, unknown: refund
pasword rest                          none      unknown: pasword rest
Can I get my money back if I cancel?  none      unknown: back cancel money
what about it?                        none      short, points back
What about the second one?            D4 (1)    points back, unknown: second

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Stop dropping stop words

In the Python tab, make content_words keep every word: change its last line to return {word for word in tokenize(text)} (no stop-word test). Run.

terminal

$ Run

── what you'll see ──

question best hit problems

What does the monthly pass cost? D8 (3) unknown: does what

what about it? D5 (1) points back, unknown: what

What about the second one? D3 (1) points back, unknown: second what

Myth vs fact

Myth

If the index is good, the retriever will cope with any question.

Fact

A retriever compares the text it is given. A question with no usable words, or the wrong words, cannot be rescued by a better index.

Myth

Users should learn to ask better questions.

Fact

They will not. Real systems are measured on the questions people actually type, including the lazy ones.

Myth

A question that returns results was understood.

Fact

The word "one" in "What about the second one?" retrieves a helmet passage. A hit is a match, not an understanding.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Log the raw question, every rewritten query, the hits and the scores for each request. When an answer is wrong, the first thing to check is what was actually searched. Without the log you will blame the model.
▸
Questions follow a long tail: a few phrasings are very common, and thousands appear once. A rule or a cache covers the head cheaply, and a model call can be kept for the tail.
▸
Weak-question rates differ by channel. Voice and mobile typing give short, misspelt queries. A search box on a documentation site gives keywords. A chat window gives pronouns. Measure per channel before choosing tools.

Remember this

1
Why this phase exists. Phases 1 to 6 improved the *index side*: how passages are split, embedded, stored and ranked. But the retriever receives one more input that nobody tuned: the query, the exact text the user typed. People type "hours", "refund", "it broke", "what about the second one?". They misspell words, use their own vocabulary, and ask two things at once. No index can compensate for a query that does not carry the information needed to find the answer.
2
Five ways a question is weak. (1) Too short: "hours" could mean opening hours or the 2-hour cancellation window. (2) Points back: "what about it?", "how much is that one?" mean nothing alone, because they depend on earlier messages. (3) Misspelt: "pasword" matches no passage by keyword and lands in an unreliable place by meaning. (4) Wrong vocabulary: the user says "refund" or "money back", the handbook says "refunded in full". Topic 1.6 showed this for keywords. (5) Compound: "what does the pass cost and do I need a helmet?" has two answers in two passages, and one search often finds only one.
3
A weak question often looks like a working one. In the example below, the single word "hours" retrieves the refunds passage (it contains "2 hours"), and "What about the second one?" retrieves the helmet passage, only because that passage contains the word "one" ("must wear one"). Nothing errors. The retriever returns *something*, the model answers from it, and the user sees a confident wrong answer. This is the silent failure of Topic 0.4 again, now caused by the input.
4
Cheap detection comes first. Before adding any model call, look for the weak shapes with rules: fewer than two content words, a pointer word (it, that, one), words that appear in no passage at all. The example prints these flags next to the best hit. In production, log the flags with every question: the share of questions with a flag tells you whether a query layer is worth building at all (Topic 7.9 has the decision list).
5
Where the fix goes. A query layer sits between the user's text and the retriever. It takes the raw question (and, in chat, the conversation so far) and produces one or more better queries, a route, and sometimes filters. The retriever and the index stay untouched, which is why this phase can be added to AskDocs without moving anything built earlier. The raw question is always kept: a rewrite can be wrong, and the original is the fallback.
6
Every fix costs something. A rule is free and predictable but narrow. A model call is flexible but adds latency (hundreds of milliseconds), money, and a new way to be wrong. The rest of the phase goes from the cheapest tool to the most expensive: rules for cleaning (7.2), expansion (7.3), several queries (7.4), conversation (7.5), routing (7.6) and filters (7.7). The rule for all of them is the same: keep a step only if it raises the hit rate on your own questions by more than it costs.

Explain it without notes

Why can a perfect index still return the wrong passage?

Name the five ways a question can be weak and give an example of each.

Why does a weak question often look like a working one?

Why do cheap rules come before model calls in a query layer?

Practice

Write blind(questions, docs) that returns the questions sharing no content word with any document. Ignore the stop words {"i", "the", "a", "do", "how", "my"}. Print the blind questions for three questions of your own over two short documents.

Write looks_weak(question) returning True when the question has at most one content word or contains a pointer word (it, that, one). Print yes or no for four questions.

Trade-offs

↔
Rule-based detection: free and predictable, but it misses weak questions that happen to share a word with the wrong passage.
↔
Logging every rewrite: more storage and privacy care, but it is the only way to debug a wrong answer.
↔
A query layer at all: more moving parts and latency, in exchange for fixing the input side that the index cannot.

Done when you can

Done when you can name the five weak shapes of a question and give an example of each.
Done when you can explain why a weak question can return a confident wrong answer.
Done when you can write rules that flag short, pointing-back and unknown-word questions.
Done when you can say where the query layer sits and what it must never change.

Command Palette

Words you'll meet

Step by step

01Step 1: the index is fine, the question is not

02Step 2: a question can retrieve the wrong passage

03Step 3: flag the weak shapes with rules

04Step 4: read the table

05Step 5: where the layer sits

06Step 6: decide with numbers

Try it yourself

Predict the hit

Add your own weak questions

Open the table of what is unknown

Code & diagrams

Break it on purpose

Stop dropping stop words

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can