Topic 8.5
Hot Partitions
In one line
When one shard drowns in traffic while siblings idle — the celebrity/whale/one-tenant problem.
Think of it like this
A single checkout counter suddenly serving a huge celebrity's fan queue while the counters next to it stand empty. All the traffic piles onto one till instead of spreading across the store.
Key ideas
- 01
Cause: skewed key distribution (a celebrity user, one giant enterprise tenant, a viral post).
- 02
Symptom: one shard at 99% CPU, others at 5%; p99 latency blows up on the hot shard.
- 03
Fixes: hash-salted keys (append a random suffix for writes), sub-sharding the hot key, or split-by-content for the hot entity.
- 04
Secondary index problem: 'list all posts by celebrity' needs the hot key — the salt breaks that query; keep a lookup table.
- 05
Trick used in practice: split hot tenants into multiple logical tenant-ids routed to different physical shards, all presented as one tenant.
- 06
Interview move: volunteer 'this key will be hot, so I'll shard the hot entity across sub-shards' BEFORE being asked.
Explain without notes
Walk the celebrity-writer case: which shard dies, which fix keeps the timeline query working?
Practice
Design the sub-shard map for a hot user and the 'by user' lookup table that keeps reads correct.
Trade-offs
- ↔
Salting sacrifices single-shard reads of the whole set; the lookup table is the tax you pay back.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can identify a hot key and prescribe the salt or sub-shard fix.