Partitions and Ordering

In one line

A partition is one ordered, append-only log; Kafka guarantees order only within a partition. Records with the same key go to the same partition, so keying by entity ID (for example orderId) keeps each entity's events in order while different entities are processed in parallel.

0/5 · 0%

Think of it like this

Supermarket checkout lanes. Within one lane, customers are served in order. Across lanes, there's no order: someone in lane 3 may finish before someone who joined lane 1 earlier. If a family must be served in sequence, they queue in the same lane.

Key ideas

01
Each partition is written by appending; each record gets the next offset. Consumers read a partition sequentially, so within a partition, the order in which records were appended is the order in which they're consumed.
02
Keyed records: the default partitioner computes murmur2(keyBytes) % numPartitions. Same key, same partition, as long as the partition count doesn't change.
03
Unkeyed records: since Kafka 2.4 the sticky partitioner fills a batch for one partition before switching (Kafka 3.3 improved it to balance by broker load), which gives better batching than strict round robin. There is no ordering relationship between unkeyed records.
04
Parallelism: each partition is consumed by one consumer in a group, so partitions set the upper limit of consumer parallelism, and they spread storage and write load across brokers.
05
Ordering can still break inside a partition if producers retry with multiple in-flight requests and idempotence off (Topic 2.4), or if a consumer processes records from one partition concurrently (Topic 3.5).

Code & diagrams

key-routing.mermaiddiagram

Rendering diagram…

OrderEvents.javajava

ProducerRecord<String, OrderEvent> record =
    new ProducerRecord<>("commerce.orders", event.orderId(), event);   // key = orderId
record.headers().add("eventType", event.type().getBytes(UTF_8));
producer.send(record, (meta, ex) -> {
    if (ex == null) log.info("order={} -> partition={} offset={}", event.orderId(), meta.partition(), meta.offset());
});
// order-123 Created -> partition=2 offset=41
// order-123 Paid    -> partition=2 offset=42
// order-123 Shipped -> partition=2 offset=57

Interview problem

The problem

Guarantee order for each order's lifecycle

Order 123 emits Created, Paid, Shipped, Delivered. How do you guarantee consumers see them in that order? Explain why key = orderId works and what happens if events are spread across partitions.

The interviewer follows up

Different services produce Created (order service) and Paid (payment service). Is order still guaranteed?

Explain it without notes

Why does Kafka only guarantee ordering within a partition?

Practice

Produce 10 events for 3 keys to a 6-partition topic and verify each key's events are in one partition in order.

Trade-offs

↔
Finer keys (per order) spread load well but only order within that entity; coarser keys (per customer) order more events together but risk hot partitions.

Done when you can

I can explain partition-scoped ordering and use keys to get per-entity ordering.

Command Palette