Topic 6.4
Ordering End to End: What Kafka Guarantees and What You Must
In one line
Kafka orders records within a partition. End-to-end order also needs the right key, one producer path per key with idempotence, consumers that process each partition (or key) sequentially, and retry strategies that don't reorder. Global ordering across everything requires a single partition, which caps throughput.
Think of it like this
A single-file corridor. People leave in the order they entered only if there's one corridor, nobody steps aside to tie a shoelace (retry topic), and the doorman lets them out one by one (sequential consumer).
Key ideas
- 01
Producer: same key → same partition; idempotence keeps order across retries with up to 5 in-flight requests; different producers writing the same key can interleave in wall-clock arrival order.
- 02
Broker: append order is the order within the partition, full stop.
- 03
Consumer: one partition is read in order by one group member; parallel processing within the partition must be per-key sequential; non-blocking retry topics move failed records out of order.
- 04
Global ordering: all events in one sequence means one partition (one leader, one consumer thread). That can handle tens of MB/s at best, not a large system's full load. Alternatives: order per entity (usually what's really needed), or attach sequence numbers or timestamps and re-sort downstream in bounded windows.
- 05
Cross-topic ordering doesn't exist: events in different topics have no defined relative order. Put causally related events on the same topic and key, or carry causal metadata (version numbers) so consumers can detect and handle out-of-order arrival.
Code & diagrams
Interview problem
The problem
Global ordering for all users' events
A stakeholder requires that all events from all users be globally ordered. Can Kafka do this while keeping high partition-level parallelism? Explain the trade-off and what you'd propose.
Explain it without notes
List the ways ordering can break even when all events for a key go to one partition.
Practice
Design ordering for a chat app: messages in one conversation must be ordered; conversations are independent.
Trade-offs
- ↔
Wider ordering scope means fewer parallel lanes and lower throughput.
Done when you can
I can explain end-to-end ordering requirements and the global-ordering trade-off.