PHASE 6Intermediate ~41 min· topic 5 of 8

Topic 6.5

Long Answers and Performance

In one line

Parsing and drawing the whole answer again on every token costs more as the answer grows. Keep finished blocks as they are, parse only the paragraph still being written, draw only the part of a long chat that is in view, and measure before you tune.

Think of it like this

Someone reads you a letter aloud, and each time the writer adds a sentence at the end, the reader goes back to the first line and reads the whole letter again. The first lines are the same every time. Re-parsing a streamed answer from the top on every token is that reader.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Parse: Read text and build a structure from it. Here: Markdown text to blocks.
React.memo: A wrapper that makes React skip a component when its inputs did not change.
Long task: JavaScript that blocks the page for more than about 50 ms. Users notice it as stutter.
Throttle: Allow an action at most once per time interval, and ignore or batch the rest.
Virtualisation (windowing): Drawing only the rows of a long list that are in view, and keeping the scroll box at full height.
Overscan: A few rows drawn beyond the visible edge so fast scrolling never shows a blank gap.
Profiler: A tool that records how long each component took to render.
Quadratic growth: Total work that grows with the square of the size: doubling the text makes the total work four times larger.

Step by step

01Count the work

Timings differ from machine to machine, so the first example counts instead: characters handed to the parser, and times a block component ran. It streams a 1,052-character answer (12 blocks) in 176 updates of 6 characters each.

The plain version parses 93,452 characters in total, about 89 times the answer's length, and runs a block component 1,153 times.

02Skip blocks that did not change

The second run wraps the block in memo, with a check on a signature of its content. The parser still gets all 93,452 characters, but the block component runs only 182 times: once per update for the block being written (176 updates), plus one extra run in each of the 6 updates where the new text ended one block and began the next, so two blocks changed. (A run counted 170 updates with one render and 6 with two.)

03Parse the new part only

The third run caches the parse of every finished paragraph and parses only the last one. Parsed characters fall to 9,405: about nine times the answer's length instead of eighty-nine. Rendering stays at 182.

Parse the new part onlydiagram

Rendering diagram…

04A chat with 2,000 messages

The second example draws a 2,000-row chat in two ways. Drawing every row puts 2,000 elements into the page. The windowed version draws 10 (8 in view, 2 of overscan), and after jumping to message 1,001 it draws 12, numbers 999 to 1010.

Every row is 40 pixels tall here. Real messages have different heights, which is why real projects reach for a virtualisation library.

05An answer nobody reads to the end

The third example takes a 27,190-character answer and draws the first part, cut at a paragraph boundary, with a button that says how much more there is. Drawing 90 paragraphs instead of 600 keeps the first paint fast, and a click draws the rest.

06Use the SDK's throttle and your browser's tools

useChat({ throttle: 50 }) batches message updates to at most one every 50 ms (the older name experimental_throttle still exists but is deprecated). Open the Performance panel, record a long answer with 4x CPU slowdown, and look for red long-task markers. Add performance.mark calls around your parse step if you need numbers on a real device.

Try it yourself

1
Change the chunk size
In the first example, change both 6s in run to 1 and run it again.
A run printed {"chunks":1052,"parsed":553878,"renders":6840} for the plain version: about 526 times the answer's length, against 89 times with six characters per update. That is the quadratic growth. The incremental version printed parsed 51,137 (about 49 times the answer), far smaller but not flat: each update still parses the paragraph being written, so the cost is now the length of one paragraph per update, not of the whole answer.
2
Shrink the window
In the second example, set VIEW = 80 and OVERSCAN = 0, and scroll by hand in the preview.
Rows appear and disappear at the edges, with brief gaps when you scroll fast. Overscan is what hides that.
3
Cut in the wrong place
In the third example, replace the body of firstPart with return text.slice(0, limit); and run it.
The last paragraph on screen is cut in the middle of a word (a run showed Sect). Cutting at a paragraph boundary keeps the sentence whole.

Code & diagrams

Counting work: parse everything, skip finished blocks, parse only the new part

These are counts, not timings, so they are identical on every machine. The updates are applied one after another with flushSync so React cannot merge them.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

answer length: 1052 characters, 12 blocks
re-draw every block on every update: {"chunks":176,"parsed":93452,"renders":1153}
skip blocks whose text did not change: {"chunks":176,"parsed":93452,"renders":182}
also parse each finished paragraph only once: {"chunks":176,"parsed":9405,"renders":182}

A chat with 2,000 messages: draw all, or only what is in view

Scroll the box with your mouse and watch the numbers in the rows change: the page never holds more than about a dozen. The waits in this example are not racing a stream, they let React draw after the scroll event.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

draw all: 2000 rows in the page
windowed, at the top: 10 rows, from Message 1 to Message 10
windowed, after scrolling: 12 rows, from Message 999 to Message 1010

An answer too long to draw at once

A real product would also fold very long code blocks and keep the full text out of any code that copies the message list on every update.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

answer: 27190 characters, 600 blocks
drawn at first: 90 paragraphs
button: Show the rest (23196 more characters)
after clicking: 600 paragraphs, button gone: true

Throttle, defer and measure (not live)tsx

useChat's throttle option is in the installed @ai-sdk/react 4.0.141 types; @tanstack/react-virtual is at 3.14.19 on npm. content-visibility is plain CSS.

// 1. Batch the SDK's updates: at most one re-render every 50 ms while streaming.
const { messages } = useChat({ throttle: 50 });

// 2. Keep typing responsive: urgent updates first, the Markdown update when React has time.
const deferredText = useDeferredValue(part.text);
<Markdown text={deferredText} />;

// 3. Put numbers on it (use the Performance panel with 4x CPU slowdown too).
performance.mark("md-start");
const blocks = parseBlocks(text);
performance.mark("md-end");
console.log(performance.measure("markdown parse", "md-start", "md-end").duration);

// 4. Skip layout work for rows that are off screen (CSS only).
//   .message { content-visibility: auto; contain-intrinsic-size: auto 120px; }

// 5. Real virtualisation with variable heights: @tanstack/react-virtual
import { useVirtualizer } from "@tanstack/react-virtual";
const virtualizer = useVirtualizer({ count: messages.length, getScrollElement: () => scrollRef.current, estimateSize: () => 96, overscan: 4 });

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

A memo that compares the wrong thing

In the first example, change the comparison in memo from prev.sig === next.sig to prev.b === next.b (compare the block objects) and run it.

terminal

$ The console

── what you'll see ──

re-draw every block on every update: {"chunks":176,"parsed":93452,"renders":1153}

skip blocks whose text did not change: {"chunks":176,"parsed":93452,"renders":1153}

also parse each finished paragraph only once: {"chunks":176,"parsed":9405,"renders":186}

Myth vs fact

Myth

React is fast enough that I never need to think about this.

Fact

React is fast at drawing what changed. If you ask it to redraw 400 blocks per token, it will.

Myth

useMemo or memo everywhere makes an app fast.

Fact

Memoise where you measured wasted work, as with finished blocks here. Everywhere else it adds code and memory for no gain.

Myth

Virtualisation is always the right fix for long lists.

Fact

It removes elements from the page, and with them find-in-page and screen-reader access to off-screen rows. Try content-visibility: auto first.

Myth

If it is smooth on my laptop it is smooth.

Fact

Phones are several times slower. Throttle the CPU in DevTools and test there.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Stable keys decide whether React reuses an element. Use the block's position while streaming, not a key made from its text, or every character remounts the element.
▸
Auto-scroll deserves care. Scroll to the bottom as text arrives only if the user is already at the bottom; if they scrolled up to read, leave them there and show a "jump to latest" button. Remember to stop the stick-to-bottom behaviour during onScroll events you caused yourself.
▸
Memory also grows: a long chat keeps every message and its parts in memory. For very long sessions, store old messages on the server and load them in pages when the user scrolls up.
▸
Your rate of rendering also affects battery and fans. A throttled stream is kinder to a laptop on the sofa than one that redraws 300 times a second.

Remember this

1
The problem this solves: In Topics 6.1 to 6.4 the page re-parses and re-draws the whole text for every new piece. For a 100-word answer nobody can tell. For a 2,000-word answer arriving a few characters at a time, the work for each update grows with the length of the text, so the total grows with its square. The page starts to stutter exactly when the answer is longest. And a chat with thousands of messages makes a different kind of slowness, in the page itself.
2
Where the time goes. Four places. *Parsing* the Markdown text. *React* running a component for every block. The *browser* updating the page and re-laying it out. *Highlighting* code (Topic 6.2). A fifth appears in long chats: the sheer *number of elements* on the page. Each has its own fix, so find which one hurts before you fix any.
3
Measure first. Use counts you can reason about (characters parsed, components run, elements on the page), the React DevTools Profiler, performance.mark() and performance.measure(), and the browser's Performance panel. A long task is any piece of JavaScript that blocks the page for more than about 50 milliseconds; users feel them as stutter. Test on a slow phone or with the CPU throttled in DevTools, because your laptop hides the problem.
4
Keep finished blocks. Once a paragraph is followed by a blank line, it will not change. Give each block a stable key (its position) and wrap the block component in React.memo with a check on the block's content. React then skips every finished block and only the last one runs. The first example shows 1,153 block renders fall to 182 for the same stream.
5
Parse only the unfinished part. Split the text at blank lines. Parse each finished piece once and keep the result (a cache keyed by the piece's text). Parse only the last piece on each update. In the example the characters handed to the parser drop from 93,452 to 9,405. The simple version breaks when a code block contains a blank line, because the split falls inside the fence: real libraries track fence state, and so must you if you go this way.
6
Do not render more often than the screen can show. A screen refreshes about 60 times a second, roughly every 16 milliseconds. If tokens arrive every 3 milliseconds, most renders are never seen. useChat accepts throttle (a number of milliseconds) to batch its updates. React's useDeferredValue and startTransition let urgent work (typing in the composer) run before the Markdown update. Pick a throttle by measuring, commonly 30 to 100 ms.
7
Long chats: draw only what is in view. This is called virtualisation or windowing. The scroll box gets the full height (rows times row height), and only the rows in view, plus a few extra (the overscan), are drawn, placed with position: absolute. In the second example, a 2,000-message chat has 10 rows in the page instead of 2,000. Variable-height rows need measuring (use a library such as TanStack Virtual, 3.14.19), and the chat must stay stuck to the bottom as new text arrives unless the user scrolled up.
8
Costs of virtualising. The browser's find-in-page (Ctrl+F) cannot find text that is not drawn. Screen readers only know the rows that exist. A simpler first step is the CSS property content-visibility: auto with contain-intrinsic-size, which lets the browser skip the layout work of rows that are off screen while keeping them in the page.
9
One enormous answer. Some answers are far too long to read in one go. Draw the first part (cut at the end of a paragraph, not in the middle of one) and offer "Show the rest", as the third example does: 90 paragraphs first, 600 after the click. Also keep the full text out of places that copy it on every update, such as a deep clone of the message list.

Explain it without notes

Why does re-parsing the whole answer on every token get slower as the answer grows?

How does React.memo help a streaming message, and what must be true for it to work?

What is virtualisation, and what do you lose by using it?

How do you decide where to optimise a slow streaming UI?

Practice

In the first example, add a fourth run, Throttled, that applies an update only every 5th chunk, and compare its counts with the others.

Write a hook useStickToBottom(ref, deps) that scrolls a box to the bottom when deps change, but only if the user was already at the bottom.

Trade-offs

↔
Caching parsed paragraphs saves a lot of work and adds state that must be reset when the message changes or is regenerated. A stale cache shows an old paragraph, which is a worse bug than slowness.
↔
A throttle of 100 ms is cheap on the CPU and makes the stream look chunky. 30 ms looks smooth and costs more. There is no right answer except the one you measure on your slowest device.

Done when you can

I can explain why re-parsing everything on every token grows with the square of the length.
I can skip finished blocks with a stable key and memo.
I can parse only the unfinished paragraph, and name the code-fence trap in doing so.
I can draw only the rows of a long chat that are in view.
I can measure with counts, the Profiler or the Performance panel before I tune.

Command Palette

Words you'll meet

Step by step

01Count the work

02Skip blocks that did not change

03Parse the new part only

04A chat with 2,000 messages

05An answer nobody reads to the end

06Use the SDK's throttle and your browser's tools

Try it yourself

Change the chunk size

Shrink the window

Cut in the wrong place

Code & diagrams

Break it on purpose

A memo that compares the wrong thing

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can