PHASE 10Intermediate ~26 min· topic 2 of 8

Topic 10.2

Showing Uncertainty Honestly

In one line

A model gives no reliable measure of how sure it is, so the screen must tell the truth in other ways: say the text is AI-written, show the sources it rests on, say when there are none, note how an answer ended, and ask for a check only where a mistake costs something.

Think of it like this

A new shop assistant. A good one says "I'm not sure, let me check the file" when they don't know, and points at the page when they do. A bad one says everything in the same confident voice. Customers trust the first kind, because their confidence means something.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Hallucination: A fluent answer that states something false or invented. The model does not know it is doing so.
Source: A page or document an answer was based on, sent to the browser so the user can open it.
source-url: The AI SDK message part type for a source that is a web address. It has url and title.
Finish reason: Why the model stopped: stop (done), length (hit the limit), content-filter (a safety system stopped it), and others.
Calibration: Whether stated confidence matches how often the answer is right. UI labels should not pretend to be calibrated.
Label noise: Warnings so frequent that users stop reading them, including the ones that matter.
Verify: Check an answer against something that is not the answer, such as the policy page or the order record.

Step by step

01Decide what your system actually knows

Write down the facts your code can see: did retrieval return documents? How did the response end? Does the answer contain an amount or a date? Everything on the label must come from one of those facts. Nothing else is honest.

02Send sources with the stream

The first example's server uses createUIMessageStream. It writes one source-url part for every page it found, then merges the model's own stream into the same response. The browser receives both in order, and useChat puts the source parts into the assistant message's parts.

In a real app the pages come from your search, and the same code runs for any number of sources, including zero.

server (shape of the code)whole filets

const stream = createUIMessageStream({
  execute: async ({ writer }) => {
    for (const page of found) {
      writer.write({ type: "source-url", sourceId: page.id, url: page.url, title: page.title });
    }
    const result = streamText({ model, messages: await convertToModelMessages(messages) });
    writer.merge(result.toUIMessageStream());
  },
});
return createUIMessageStreamResponse({ stream });

03Turn facts into one label

trustLabel(message) counts the source parts and picks a sentence. Two cases, two sentences. Notice what it does not do: no scores, no colours that imply a grade, no word "verified".

04Handle every ending

The second example asks the same stand-in model for three kinds of ending. finishReason is stop, length or content-filter, and endingNote gives each a plain sentence. A normal ending gets no note at all, because a note on everything is noise.

Handle every endingdiagram

Rendering diagram…

05Put the check where it counts

The third example is a function from answer text to a note. It looks for amounts (a currency sign and digits) and time limits (a number and "days"), and it removes order ids first, because the digits in ORD-1042 are not an amount. Of six sample answers, three get a note. Run the example, then widen the rule to see how quickly it starts warning about everything.

06Keep labels calm and consistent

Same place, same words, small size. Use colour only as a second cue (never the only one; Topic 10.6 covers contrast). When a label changes, the user should be able to guess why from its words.

Try it yourself

1
Add a second source
In the first example add a second row to PAGES that also matches "return", for example { match: /return/i, url: "https://chai.example/help/refunds", title: "Refunds" }. Run it.
The label becomes "Based on 2 pages" and the page shows two links. The label code did not change, because it reads the parts and not a fixed number.
2
Add a new ending
In the second example add ["a broken ending", { text: "Opened tea", error: "connection lost" }] to the list and run.
The call throws an error instead of returning a finish reason, and the loop stops. A failure is not a finish reason: it needs its own path (Topic 3.3 covers a broken stream and partial answers).
3
Widen the amount rule
In the third example change MONEY to /\d[\d,]*(?:\.\d+)?/ (any number) and run.
The "7 days" answer is now told to check "the exact amount", which is wrong. Loose rules produce wrong warnings, and wrong warnings teach people to ignore the right ones.

Code & diagrams

Sources and an honest label, including when there are none

The server found a page for the first question and none for the second. The only thing that changed is a fact the server knew. The second question is scripted for you. Note that a source label says what the answer rests on, not that it is correct.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"Unopened items can be returned within 7 days of delivery." -> Based on 1 page from Chai & Co
"I think we might ship there, but I am not sure." -> AI-written, no source found. Check with our team before you rely on it.

No confidence number, and a note for every ending

The first line is empty on purpose: the result has no field that says how sure the model is. What it does have is how the answer ended. The `length` and `content-filter` endings are produced by the stand-in model; real providers report them the same way.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

keys that look like confidence: []
finishReason: stop | has usage: true
normal ending        | stop           | ""
hit the length limit | length         | "This answer was cut short because it hit the length limit. Ask me to continue."
stopped by a filter  | content-filter | "I can't help with that one. Try asking in a different way."

A note only where a mistake costs something

The rule is a heuristic: it catches the amount and the day count, ignores the order id and the empty answer, and it will miss some things. The note is a nudge for the user, not a safety measure.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

"Masala chai is black tea boiled wi"   -> ""
"Order ORD-1042 has shipped."          -> ""
"Unopened items can be returned wit"   -> "Check the date in our policy page."
"Your refund of ₹450 will arrive in"   -> "Check the exact amount on your order page."
"Shipping costs Rs. 1,200.50 for th"   -> "Check the exact amount on your order page."
""                                     -> ""
answers with a note: 3 of 6

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Treat every number as an amount

In the third example replace the MONEY line with const MONEY = /\d[\d,]*(?:\.\d+)?/; and run.

terminal

$ The console, third line

── what you'll see ──

"Unopened items can be returned wit" -> "Check the exact amount on your order page."

Myth vs fact

Myth

A confidence percentage helps users trust the answer.

Fact

Only if it is real and calibrated. The SDK result has no such field, and a number you invent gives false precision. Show what you actually know: sources, the ending, whether anything was found.

Myth

Showing a source means the answer is correct.

Fact

The model can misread or misquote its source. A source lets the user check; word the label as a description ("Based on…") and not as a guarantee.

Myth

More warnings make the product safer.

Fact

Warnings on everything are ignored, including the one that matters. Use few, specific notes where a mistake costs something.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Cite in the text, not only under it. Where the answer makes a claim, a small numbered marker that links to its source lets the user check one sentence at a time. This needs the model to emit the markers and your code to validate that each marker refers to a real source; unchecked markers are as unreliable as unchecked links (Topics 6.4 and 6.6).
▸
Make the "no source" state a product decision, not an accident: some products refuse to answer without a source, some answer with the label. Either is honest. Silence about it is not.
▸
Log how often answers have no source and how often users click a source. The first number tells you where your content has holes; the second tells you whether sources are noticed or ignored. Both are cheap and neither needs the answer text (Topic 9.7 on what not to log).

Remember this

1
The problem this solves: A model writes a wrong answer in exactly the same fluent tone as a right one, and the response carries no flag (Topic 0.1, fact 4). If your screen also looks equally sure about everything, people either believe wrong answers or learn to believe nothing. This topic gives you honest signals you can really build.
2
You cannot show a confidence percentage. Some apps print "92% sure". The numbers the SDK returns do not say that. The first live example looks at a real result: its keys include finishReason and usage, and none of them is a confidence score. A percentage you make up (or compute from something unrelated, like the length of the answer) is a false signal and worse than none. Model-side ideas such as token probabilities exist, but they measure how likely words were, not whether a claim is true (see /llm-engineering).
3
What you can honestly show, part 1: what it is. A small, steady label that the text is written by AI. Keep it in the same place for every answer. The goal is context, not a warning sign: a short "AI-generated" near the message or in the composer is enough. Large red banners on every message become wallpaper.
4
Part 2: what it rests on. If your server fetched pages or documents before answering (retrieval, taught in /rag), send them to the browser as sources. In the AI SDK a source is a message part of type source-url (with url and title) or source-document. Show them as links under the answer. Sources let the user check; they do not prove the answer is right. A model can cite a page and still misread it, so word the label as "Based on 1 page" rather than "Verified".
5
Part 3: say when there is nothing. The second answer in the first example had no source. The label changes to "AI-written, no source found. Check with our team before you rely on it." Honest absence is the most valuable signal you have, and it costs nothing: your server knows whether retrieval found something.
6
Part 4: how the answer ended. Every response ends with a finish reason. stop is normal. length means the answer hit the limit and may stop mid-sentence. content-filter means the provider's safety system stopped it (Topic 9.6). A cut-off answer that looks complete is a quiet form of lying, so give each ending its own note. The second example does that with real results.
7
Part 5: ask for a check, only where it matters. A "please verify" note on every message is noise, and people stop reading it. Put it where a mistake costs money or time: amounts, dates, deadlines, medical or legal details. The third example is a small, testable rule for that. It is a heuristic and will miss things, so treat it as a nudge and not a safety guarantee (real safety belongs on the server: never let the model's number be the number you charge).
8
Never let wording promise more than the system knows. "Verified", "accurate" and "official" are claims. "Based on", "AI-written" and "check before relying" are descriptions. Phase 9 explained why a model's words can be steered; honest labels keep your product from vouching for that.

Explain it without notes

Why should an AI feature not show a confidence percentage?

What facts can the frontend honestly show about an answer?

Why does Mitra give notes only on some answers?

What should the screen do when an answer ends with finishReason: "length"?

Practice

Change trustLabel so that when there are two or more sources it says "Based on N pages. Open them to check." and for exactly one it says "Based on 1 page." Log both.

Write endingNote for the finish reason "tool-calls" for a case where the stream stopped while a tool was still running, and log it.

Trade-offs

↔
Labels and notes add words to every screen. A calm, consistent amount is respectful; more is noise. Decide the rule once and test it on real answers.
↔
Showing sources requires a retrieval step, which has its own cost and failure modes (/rag). Without one, "AI-written" and the ending notes are still honest, just less helpful.

Done when you can

I can explain why Mitra shows no confidence percentage.
I can send sources with the stream and read source-url parts in useChat.
I can write a different label for "has sources" and "no source found".
I can show a note for length and content-filter endings.
I can limit verify-notes to answers where a mistake costs something, and say why.

Command Palette

Words you'll meet

Step by step

01Decide what your system actually knows

02Send sources with the stream

03Turn facts into one label

04Handle every ending

05Put the check where it counts

06Keep labels calm and consistent

Try it yourself

Add a second source

Add a new ending

Widen the amount rule

Code & diagrams

Break it on purpose

Treat every number as an amount

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can