Topic 11.6
User Feedback Loops
In one line
Thumbs up and down, with a reason, tell you what automatic metrics cannot: whether the answer was useful. Join each vote to the request that produced it, compare prompt versions, and send the bad answers into your golden set.
Think of it like this
A restaurant with a comment card on each table. Kitchen counts (plates sent out, minutes per order) tell the manager the kitchen is busy. Only the cards tell her that the soup was cold. And a card that says "cold soup, table 7, Tuesday" is useful because she can find that order and fix the recipe.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Feedback
- A signal from the user about a reply, such as a thumbs up or down and a reason.
- Request id
- The id that ties a vote to the exact request and reply it is about.
- Join
- Combining two lists of records by a shared key, here feedback and requests by request id.
- Golden set
- A fixed set of test cases with expected results, run before every change (Topic 9.2).
- Selection bias
- The people who vote are not a typical sample of all users.
- Prompt version
- An id for the exact prompt text used, recorded on every request.
Step by step
01Step 1: log what you need to join on
Each request record keeps its request_id, prompt version and question. The UI shows the id to the front end, and the thumbs buttons send it back with the vote. If the id is missing at either end, feedback cannot be used.
02Step 2: collect small structured votes
The vote has three fields: request id, up or down, and a reason from a short list. Only the thumbs-down shows the reason picker, to keep the happy path to one click.
feedback = [
("r3", "down", "wrong_answer"),
("r8", "down", "too_long"),
("r10", "up", ""),
]03Step 3: join and compare versions
For each vote, look up its request and add the vote to that request's prompt version. The output compares v1 (3 up, 1 down) with v2 (1 up, 3 down). A drop like this after a prompt change is a signal to look, not yet a proof.
04Step 4: count reasons
Group the thumbs-down by reason and sort by count, with the reason name as a tie-breaker so the order is stable. Three of four are wrong_answer; that is a quality problem, not a style problem.
05Step 5: collect the wrong answers for review
List the requests with wrong_answer and print their questions. A person reads each one, decides what the right answer is, and adds it to the golden set. Never add them automatically: some votes are mistakes or abuse.
06Step 6: close the loop
After the fix (a better prompt, better retrieval, a different model), run the golden set. The cases that used to fail should now pass, and the next feedback batch should show fewer thumbs-down for that reason. Without checking, you only hope.
Try it yourself
- 1
Predict before running
Before you run, count the votes by prompt version by hand. Which version has the higher share of thumbs-up, and which reason is most common?
- 2
Add a vote with an unknown request
Add the vote
("r99", "down", "wrong_answer"). What happens in your language? Decide what the code should do with a vote whose request was deleted or never logged. - 3
Add a minimum count
Change the version report to print
too few voteswhen a version has fewer than 5 votes. Why is this more honest?
Code & diagrams
12 logged requests and 8 votes. The join compares two prompt versions, counts reasons and lists golden-set candidates.
Expected output
feedback on 8 of 12 requests
v1: 4 votes, 3 up, 1 down, 75% up
v2: 4 votes, 1 up, 3 down, 25% up
reason wrong_answer: 3
reason too_long: 1
add to the golden set:
r3: Can I change my delivery date?
r7: Where is my order 5120?
r9: Can I return a gift?Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Divide by the wrong count
In the Python version, change votes = row["up"] + row["down"] to votes = 6.
Myth vs fact
Myth
Few votes means nobody cares.
Fact
Most users never vote. A handful of reasoned thumbs-down is a strong signal, and each one is a ready test case.
Myth
If thumbs-up is high, the answers are correct.
Fact
Users often cannot tell a confident wrong answer from a right one. Feedback finds usefulness and obvious errors; evaluation checks correctness.
Myth
Feedback can go straight into training or the golden set.
Fact
Votes can be mistaken or abusive and the text can hold personal data. Review, redact and get permission first.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Add implicit feedback to the explicit votes: the user re-asked the same question, copied the answer, abandoned the page, or edited the output. These signals are free and more numerous, though noisier.
- ▸
Keep the exact prompt version, model and retrieved chunk ids with each request, so a thumbs-down can be replayed against a candidate fix in an offline run.
- ▸
Separate "I did not like the tone" from "it is factually wrong". Route each reason to a different owner: prompt writers, retrieval, or product.
Remember this
- 1
Why this exists. Latency, errors and cost are easy to measure. Whether the answer was good is not. A reply can be fast, cheap, well-formed and wrong. Users are the only ones who know for sure, so ask them cheaply: a thumbs up or down on each reply, and optionally a reason. This signal catches the quiet failures that no status code shows (Topic 0.4).
- 2
What to collect. The vote (up or down), an optional reason from a short fixed list (
wrong_answer,too_long,off_topic,unsafe,other) and, optionally, a free-text comment. Fixed reasons can be counted; free text needs reading. The vote must carry the request id of the reply it is about. Without that id, a thumbs-down is only a mood. - 3
The join. Feedback becomes useful when you join it to the request record that you logged earlier (Topic 11.2): the question, the prompt version, the model, the retrieved chunks. The example joins 8 votes to 12 requests. It shows what the join gives you: v1 has 75% positive votes and v2 has 25%, so the new prompt looks worse, and the reasons show it is mostly
wrong_answer. - 4
Be careful with the numbers. Only a small share of users vote (here 8 of 12 requests have a vote, and in a real product usually only a small share does). People who vote are not a random sample: unhappy users vote more, and very happy ones sometimes do. So compare versions with the same method, not with the absolute percentage, and look at the counts too: 3 of 4 is not statistically different from 2 of 4. Use feedback to find examples and trends; confirm with your evaluation set (Phase 9).
- 5
Feed the golden set. Every thumbs-down with
wrong_answeris a candidate test case. A person reviews it and writes the expected answer or the rule it should satisfy. Then it goes into the golden set: the fixed list of cases you run before each prompt or model change (Topic 9.2). Over time the golden set contains your real failures, which is more valuable than invented cases. - 6
Privacy and consent. A feedback comment can contain personal data, and a vote links text to a user. Redact comments, keep them under the same retention rules as logs (Topic 11.2), and tell users what happens to the text. Do not train on or share feedback without clear permission.
Explain it without notes
Why must feedback carry a request id, and what do you do with it?
What are the weaknesses of thumbs up and down as a quality signal?
How does feedback turn into regression tests?
Practice
Write score(votes) for a list of "up"/"down" strings that returns a string like "3 of 4 up (75%)" (whole percent, rounded down) or "no votes".
Write join_feedback(requests, feedback) where requests is a dict of id to question and feedback a list of (id, vote); return the questions of the thumbs-down in input order, skipping votes for unknown ids.
Trade-offs
- ↔
A simple thumbs gets many votes and little detail; a form with reasons gets detail and fewer votes.
- ↔
Showing the buttons on every reply invites feedback and adds noise; asking after key moments is cleaner.
- ↔
Using feedback quickly speeds fixes and risks over-reacting to a few votes; review before acting.
Done when you can
Done when you can name what a feedback record must contain.
Done when you can join votes to request logs and compare prompt versions.
Done when you can explain the weaknesses of the numbers.
Done when you can move reviewed failures into the golden set.