Hot Partitions and Skewed Keys

In one line

When one key (a huge customer, a viral product, a busy device) dominates traffic, its partition's broker and consumer are overloaded while others idle. Fix it by narrowing the key, bucketing hot keys with a suffix, splitting hot tenants onto dedicated capacity, or doing two-stage aggregation, each trading away some ordering or simplicity.

0/5 · 0%

Think of it like this

One checkout lane at a supermarket where a caterer shows up with 40 trolleys. Other lanes are empty, but the caterer's items must go through one lane if they need one receipt. Unless you split the order into several receipts, that lane is the bottleneck.

Key ideas

01
Symptoms: one partition's bytes-in and lag far above others; one consumer's CPU pegged; broker leader for that partition hotter than peers.
02
Composite or narrower keys: customerId|orderId instead of customerId when ordering is only needed per order.
03
Key bucketing (salting): for known hot keys, append a bucket suffix (customer-123#0..7) chosen by hash of a sub-entity or round robin. Load spreads across 8 partitions; per-customer order is lost, per-bucket order kept.
04
Two-stage aggregation: for counters or aggregates, partition by key#bucket for the first aggregation stage, then re-key by key to combine partial results (a small volume) in the second stage.
05
Dedicated capacity: route a hot tenant to its own topic or partition range with its own consumer group and resources, so it can't starve others.
06
Accept and optimise: if strict order for that key is non-negotiable, the only lever is making that one consumer faster (batching, efficient code), since you can't parallelise an ordered stream.

Code & diagrams

bucketing.javajava

static final Set<String> HOT = Set.of("customer-123");     // from config / detection
static final int BUCKETS = 8;

String partitionKey(Event e) {
    if (HOT.contains(e.customerId())) {
        // Stable per order: an order's events stay together and ordered.
        int bucket = Math.floorMod(e.orderId().hashCode(), BUCKETS);
        return e.customerId() + "#" + bucket;
    }
    return e.customerId();
}

two-stage.mermaiddiagram

Rendering diagram…

Interview problem

The problem

One partition gets 50% of all traffic

Monitoring shows partition 5 receiving about 50% of all traffic because one customer's key dominates. Consumers for other partitions are idle. What do you do?

Explain it without notes

Explain key bucketing and what it costs.

Practice

Compute: 48 partitions, one key has 30% of traffic. How much hotter is its partition than average, and how many buckets bring it within 2× average?

Trade-offs

↔
Every hot-partition fix trades ordering scope, complexity or cost for load spread.

Done when you can

I can detect hot partitions and pick the right mitigation for the ordering requirement.

Command Palette