Topic 8.6
Answer Formats
In one line
Prose is not always the right output. Quotes, steps and a validated JSON object with an answer, citations and a confidence let a product show sources and let code check the model. Streaming needs one extra care: hold back half-finished citation markers.
Think of it like this
A form versus a letter. A letter lets the writer say anything, and the reader must dig out what matters. A form has boxes: name, date, reason. The office can check it box by box. A product that shows answers with sources and acts on them wants the form.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Structured answer
- An answer with named fields, such as answer, citations and confidence, instead of free text.
- Schema
- A description of the fields an object must have and their types.
- Validation
- Checking that a value has the right shape and obeys the rules before using it.
- Quote
- An exact sentence copied from a source, checkable by string search.
- Confidence
- How sure the answer claims to be. A hint, not a measurement, unless you calibrate it.
- Chunk (stream)
- One piece of text that arrives from a streaming model. Not the same as a document chunk.
- Hold back
- Keep unfinished text in a buffer until it can be shown safely.
Step by step
01Step 1: choose the format that matches the use
Policy question: a quote plus a one-line answer. How-to: numbered steps. Anything a program reads: JSON. Chat: prose with markers.
02Step 2: describe the shape to the model
The instruction lists the three fields and their allowed values, and gives the refusal as a valid object: {"answer": "I could not find that in the handbook.", "citations": [], "confidence": "none"}.
03Step 3: parse, then validate in layers
Parse. Check types. Then check the rules only your product knows. Return a list of problems so a log line can say exactly what was wrong.
04Step 4: decide what a problem does
Retry once with the problem list appended as feedback, then fall back to the refusal. Never show an unvalidated reply.
05Step 5: stream the prose version
Chunks arrive as text. The safe emitter keeps anything from the last [ onward in a buffer until a ] appears, so the screen never shows half a marker.
06Step 6: validate at the end
When the stream finishes, parse the markers from the full text and run the citation check from Topic 8.3. If it fails, replace or annotate the answer on screen.
Try it yourself
- 1
Break the reply
Add a sixth reply with
citationsas the string"1"instead of the list[1]. Which problem message appears? Add another withcitations: [1.5]. - 2
Move a chunk boundary
Change the chunks so
]and.arrive together and[ends a chunk by itself. Predict the safe pieces before you run. - 3
Add a rule
Add a rule that
confidencemust benonewhenever the answer is the refusal sentence. Which of the five replies now fails?
Code & diagrams
Replies 1 to 5 are scripted. The validator returns every problem it finds, so one log line explains the failure.
Expected output
1: ok
2: citation out of range: 3
3: not valid JSON
4: answer is missing or empty; an answer without citations must be the refusal; confidence must be one of high, medium, low, none
5: okThe chunks are scripted to split the markers on purpose. The vertical bars show where each piece would reach the screen.
Expected output
naive: The pass costs 25 dol|lars [|1]. Rides last up to 45 minutes [|2|].
naive pieces that end inside a marker: 2
safe: The pass costs 25 dol|lars |[1]. Rides last up to 45 minutes |[2].
safe pieces that end inside a marker: 0
citations: 1,2, all valid: yesBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A range check that forgets the top
In the Python tab of the JSON example, change n < 1 or n > N_SOURCES to n < 1.
Break #2
No hold-back while streaming
In the Python tab of the streaming example, change if cut >= 0 and held.find("]", cut) < 0: to if False:.
Myth vs fact
Myth
JSON mode guarantees a correct answer.
Fact
It guarantees syntax, at best shape. Whether citations are in range and the answer is grounded is for your validators.
Myth
The model's 'high' confidence means it is probably right.
Fact
Self-reported confidence is usually not calibrated. Use scores and checks you control.
Myth
Streaming and structured output can't be combined.
Fact
They can, with care: stream prose with markers and validate at the end, or stream JSON and parse only when complete.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the format version in the prompt id and the logs. When you add a field, old cached answers and old clients must still work, so make new fields optional first.
- ▸
Quote checking is the cheapest strong grounding test: normalise whitespace and case, then test that the quote is a substring of the source. It catches invented quotes at no model cost. Allow a small edit distance only if your documents have formatting noise.
- ▸
On a validation failure, retry once with the list of problems added to the prompt. Do not loop: each retry costs money and latency, and a model that fails twice usually fails a third time (LLM Engineering Phase 6 covers retries).
Remember this
- 1
Why formats. The interface wants to show the answer, a row of source chips and perhaps a warning. A ticketing system wants a reason and a category. Code that has to parse prose will break on the first answer that is phrased differently. A structured answer gives your program fields it can validate.
- 2
Quotes. For policies and legal text, the strongest answer is often the exact sentence from the source. A quote has a property prose does not: your code can check it. If the model says the source reads 'X' and 'X' is not a substring of the source text, it made it up. A model may also be asked for a short quote and an explanation.
- 3
Steps. For how-to questions ('How do I reset my password?'), a numbered list with a citation per step is easier to follow and to check than a paragraph. The same marker rules apply (Topic 8.3).
- 4
A JSON answer. Ask for an object with
answer(text),citations(a list of source numbers) andconfidence(high,medium,lowornone). Request it with the provider's JSON mode or a schema if it has one (LLM Engineering Topics 4.2 and 4.3), and validate anyway: syntax can be right while the shape is wrong. Real Java projects use Jackson or Gson for JSON; the course'sJsonhelper keeps the Java example runnable in the browser. - 5
Validate in layers. First: does it parse? Second: are the fields there with the right types? Third, the semantic rules: every citation is between 1 and the number of sources, an empty
citationslist is allowed only for the exact refusal sentence, andconfidenceis one of the allowed words. The example returns a list of problems for five replies: a good one, an out-of-range citation, a reply wrapped in chatter, an empty one, and a correct refusal. - 6
Confidence is a hint. A model's own 'high' is not calibrated. Prefer signals you control: the retrieval score, whether citations validated, and the grounding checks of Topic 8.7. Show the model's confidence only if you have measured that it correlates with being right.
- 7
Streaming with citations. Users like to see text appear. A JSON object cannot be shown until it is complete, so streamed answers are usually plain text with
[n]markers and the sources panel arrives at the end. The trap is a marker split across chunks:[arrives in one piece and1]in the next, and the screen briefly shows a stray bracket. The second example holds back text from an unclosed[until the]arrives (LLM Engineering Topic 5.2 shows how to read the stream itself).
Explain it without notes
When would you ask for a quote, steps or JSON instead of prose?
What are the layers of validating a JSON answer?
Why is a model's own confidence not enough?
What goes wrong when you stream an answer with citations, and how do you fix it?
Practice
Write quote_ok(quote, source) that returns True when the quote appears in the source after lower-casing both and collapsing runs of whitespace to a single space. Print the result for the quote "unlimited rides" and "free rides" against "The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.".
Write next_step_ok(reply) that parses a JSON string and returns "ok" if it has a non-empty list steps of strings, else "bad" (also for invalid JSON). Print the result for '{"steps": ["Open the app", "Tap Forgot password"]}', '{"steps": []}' and 'oops'.
Trade-offs
- ↔
Prose: natural to read, hard to check. JSON: easy to check, but needs validation and cannot stream as it arrives.
- ↔
Quotes: strongest grounding, but they can read stiffly and fail on paraphrased questions.
- ↔
Strict formats: fewer surprises, more retries when the model strays.
Done when you can
Done when you can choose a format for a quote, a how-to and a program-facing answer.
Done when you can validate a JSON answer in layers and return all problems.
Done when you can explain why a model's confidence is only a hint.
Done when you can stream an answer without showing half a citation marker.