Kafka Without the Hype

Let's stop describing Kafka as 'a distributed message broker' and build it from scratch. When you can reconstruct it, you'll never need a mnemonic again.

The chain

• —Producer — the thing that has news to publish.
• —Topic — a named stream of events ('order-events').
• —Partition — a slice of the topic: an append-only, ordered log. Partitioning is how Kafka scales.
• —Broker — one server holding some partitions. A topic lives across many brokers.
• —Consumer group — a set of consumers that split the partitions among themselves.
• —Offset — a consumer's bookmark: 'I've read up to position 7 in partition 2'.

Producer ──► Topic ──► Partition (ordered log) ──► Broker
                              │
                              └──► Consumer Group (splits partitions)
                                        │
                                        └──► Offset (your bookmark)

Why partitions? Why consumer groups?

A single ordered log can only be written by one writer at a time. To write faster, split the topic into partitions and let each partition be independent — that's data parallel throughput. Consumers then split the partitions among themselves: one consumer group reading a 4-partition topic can have at most 4 active consumers; the 5th sits idle. Ordering is guaranteed per partition, not across them.

The four questions interviews actually ask

• —What happens when a consumer dies? Its partitions get rebalanced to the remaining group members. Stale offsets mean duplicates — that's why processing must be idempotent.
• —What happens when a broker dies? Partitions have replicas on other brokers; the leader fails over. Kafka prefers availability (an ISR quorum) over strict consistency.
• —How do retries work? The producer retries with acks; the consumer retries by reprocessing. Duplicates are NOT prevented by Kafka — they're handled by idempotent consumers.
• —How do I make processing idempotent? Dedupe on the natural key at the sink (e.g. upsert by orderId + eventId), or store processed offsets and skip.

Kafka gives you exactly-once as an API option, but at-least-once plus idempotent consumers is the production answer.

Consumer group sizing

Partition count is set at topic creation and is expensive to change later. Pick partitions ≥ your peak consumer parallelism and call it a deployment decision, not a configuration knob you fiddle with daily.