System 12.39 — Distributed Message Queue (design Kafka)
In one line
Design the log-based queue itself: partitioned append-only logs on disk, leader/follower replication with in-sync replicas, consumer groups with offsets, retention, and the trade-offs behind high throughput.
Think of it like this
A post office that never throws letters away after delivery. Letters for each district are kept in numbered order in separate pigeonhole rows (partitions), copied to two other branches (replication), and each reader keeps a bookmark of which letter they read last (offset).
Key ideas
- 01
Requirements: millions of messages/s, retention for days, ordering per key, at-least-once delivery (exactly-once optional), many independent consumer groups, horizontal scaling, no data loss when a broker dies.
- 02
STORAGE: each partition is an append-only log split into segment files; writes are sequential and reads use the OS page cache and zero-copy transfer. Messages are batched and compressed. Indexes map offsets to file positions. Retention deletes old segments or compacts by key (Stateful Systems course, Kafka).
- 03
REPLICATION: each partition has a leader and followers on other brokers; followers fetch from the leader; the IN-SYNC REPLICA set defines which followers are caught up;
acks=all+min.insync.replicas=2means a write survives a broker loss. A controller (Raft quorum) handles leader election and metadata. - 04
CONSUMERS: pull-based (they control their pace, a natural backpressure); consumer groups split partitions among members; committed offsets stored in an internal log. Delivery semantics come from when offsets are committed and whether producers are idempotent/transactional (Phase 10, delivery semantics).
Code & diagrams
Explain without notes
Why pull-based consumers rather than the broker pushing messages?
Practice
How does the design keep ordering while scaling consumers?
Trade-offs
- ↔
Sequential logs give huge throughput and replay but make per-message routing, priorities, and delayed delivery harder than in a traditional queue.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Stateful · Kafka fundamentals
Run Kafka, create partitioned topics, and see per-key ordering on disk.
Stateful · Kafka replication, retention & operations
ISR, min.insync.replicas, compaction, schemas, and Strimzi.
Stateful · Consumer groups & delivery semantics
Rebalancing, offsets, at-least-once, idempotency, DLQs, and consumer lag.
Completion checklist
I can design a partitioned, replicated log with consumer groups and explain its guarantees