PHASE 12Advanced ~25 min· topic 4 of 7

Topic 12.4

Showing Less to Show More

In one line

Past a point, drawing every record makes a worse chart: dots pile on dots and the page slows down. Sampling, counting in bins and zooming into detail show more truth with less drawing. The rule is never to drop data silently.

Think of it like this

A crowd photo. Two hundred thousand people in a stadium cannot each be seen. A good editor shows a few faces (a sample), or colours each stand by how full it is (counts in bins), or zooms to one section and shows faces there (detail on demand). And she writes a caption that says which one she did.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

sample: A smaller set of records chosen from the big set, for drawing.
aggregate: To combine many records into one number, such as a count per square.
bin: One group of the plane (a square or hexagon) that records are counted in.
overplotting: Marks drawn on top of each other until you cannot see how many there are.

Step by step

01Draw everything

Imagine 200,000 made-up trip starts around three busy spots drawn one fillRect each: the clusters become solid blobs and you cannot say which is denser. The first example draws that picture next to the two fixes.

02Draw a sample, and say so

A seeded sample of 2,000 shows the shape. The caption reads "Random sample: 2,000 of 200,000". Edge cases: a sample of an empty list is 0 points, and asking for more than exist gives them all.

03Count in squares

Points are counted in 10 by 10 squares on a square-root colour scale. 816 of the 200,000 are outside the 200 by 150 picture: the squares count the other 199,184 and the log says so.

04Zoom changes what you draw

The second example counts in 32-unit squares at 1x (194,851 points in view), then 16, then 8, and at 8x only 3,812 points are in view, so it draws them one by one. Its caption always names the mode.

Try it yourself

1
Bigger squares
In the first example, change const SIZE = 10; to 25. The Console says 48 squares of 48, and the most points in one square becomes 36,785. Bigger squares give fewer, smoother, less precise cells.
2
A smaller sample
Change sample(points, 2000, d3.randomLcg(1)) to sample(points, 200, d3.randomLcg(1)). The Console says sample: 200 points, 0.1% of the data, and the caption under the middle picture follows. The three clusters are still visible, thinner. Where does the sample stop showing the shape? That is your number.
3
No data at all
Change const N = 200000; to 0. The Console says points: 0 | inside the picture: 0 | outside: 0, sample: 0 points, 0% of the data and squares with at least one point: 0 of 300 | most points in one square: 0. The example guards two things: d3.max of no squares is undefined (it would throw on .toLocaleString), and 0 divided by 0 is NaN, which would print NaN%.
4
Show dots sooner
In the second example, change visible.length <= 5000 to visible.length <= 500. At 8x the 3,812 points in view are now above the limit, so the Console says zoom 8x: squares of 4, 3,812 points in view. The switch point is a number you choose, and the caption always tells the truth about which mode is showing.

Code & diagrams

All points, a sample, and counts New tab

The same 200,000 made-up points three ways. 816 of them lie outside the 200 by 150 picture, so the squares count 199,184: the first line says it.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

points: 200,000 | inside the picture: 199,184 | outside: 816
sample: 2,000 points, 1% of the data
squares with at least one point: 300 of 300 | most points in one square: 7,162
sample of an empty list: 0 | sample larger than the data: 2

Level of detail: squares when zoomed out, points when zoomed in New tab

Move the slider. The label names the mode and how many points are in view. The last two lines show a 2,000-point sample losing the one lonely trip, while its square still counts 1.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

zoom 1x: squares of 32, 194,851 points in view
zoom 2x: squares of 16, 116,827 points in view
zoom 4x: squares of 8, 16,979 points in view
zoom 8x: dots, 3,812 points in view
a 2,000-point sample contains the lonely trip: false
the square around the lonely trip still counts: 1

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Count points that are outside the picture

In the first example, replace the first argument of d3.rollup (the line points.filter((p) => p[0] >= 0 && p[0] < W && p[1] >= 0 && p[1] < H), // a point outside the picture belongs to no square) with points,.

terminal

$ Console

── what you'll see ──

points: 200,000 | inside the picture: 200,000 | outside: 0

sample: 2,000 points, 1% of the data

squares with at least one point: 336 of 300 | most points in one square: 7,162

Myth vs fact

Myth

Showing fewer points is hiding the truth.

Fact

Not if you say what you did. A caption with the counts is honest, and a blob of 200,000 overlapping dots hides more than a sample or a count map does.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
d3.rollup with string keys like "3,4" is simple but slow for millions of records. For big data compute a numeric index (i + j * columns) and count in a typed array.

Remember this

1
The problem this solves: 200,000 dots on a small canvas are mostly hidden behind other dots. The picture is a solid blob: you cannot tell dense from very dense. Reducing the data is often the clearer chart, not only the faster one.
2
Sampling. Draw a random subset: every point has the same chance, so the sample's shape follows the data's. Use a seeded generator so it is repeatable, and a proper shuffle (the example uses a partial Fisher-Yates). A sample shows where the data is, not exact counts, and can lose rare points entirely: a 2,000-point sample of the second example's 200,001 points does not contain its one lonely trip.
3
Aggregation and binning. Instead of marks for records, count records in groups. In square bins you divide the plane into squares, count the points in each and colour each square by its count. Hexagonal bins (a hexbin) do the same with hexagons, whose neighbours are all equally far away. Use a square-root colour scale (scaleSequentialSqrt) so a few huge bins do not wash out the rest. Every record is counted: the example checks that its counts add up. A count of 1 stays nearly white on that scale, so keep outliers you must not lose separately and draw them on top.
4
Never drop data silently. If a picture shows 2,000 of 200,000 points, say so on the page: "Random sample: 2,000 of 200,000". If bins left out 816 points outside the view, say that too. A reader who thinks they see everything draws wrong conclusions. Every example writes its caption from the numbers it used.

Explain it without notes

Why can a chart of 200,000 dots be worse than a chart of 2,000?

How does binning keep every record counted, while sampling does not?

Practice

Write binIndex(x, y, size, columns) that returns one number for the square holding the point, and -1 for a point outside a grid columns squares wide.

Change the second example so the lonely trip is drawn as a red dot when it is in view, even while the rest are squares.

Trade-offs

↔
Sampling is simplest and keeps the look of real points, but loses rare ones. Binning keeps every record and loses individual positions. Detail on zoom keeps both and costs more code.

Done when you can

I can say why drawing all 200,000 dots is not clearer.
I can draw a seeded sample and count points in square bins, and check the counts add up.
I know why rare points need separate care.
I never leave out data without saying how much.

Command Palette