Core Architecture: Brokers, Topics, Partitions, Replicas, Controllers

In one line

A Kafka cluster is a set of brokers that store partitions; each partition has one leader replica serving reads and writes and follower replicas copying it. A KRaft controller quorum manages metadata (which broker leads which partition). Producers write to leaders; consumers in groups divide partitions among themselves.

0/5 · 0%

Think of it like this

A chain of warehouses (brokers) storing numbered shelves (partitions) of several product lines (topics). Each shelf has a main copy in one warehouse (leader) and backups in others (followers). Head office (controller) keeps the master list of where each shelf's main copy is and reassigns it if a warehouse burns down.

Key ideas

01
Broker: a server process that stores partition data on disk and serves client requests. Topic: a named stream. Partition: an ordered log; the unit of parallelism and ordering. Offset: a record's position in its partition.
02
Replica: a copy of a partition on a broker. Leader: the replica that handles produce and (by default) fetch requests. Follower: fetches from the leader to stay in sync. ISR (in-sync replicas): followers caught up within replica.lag.time.max.ms.
03
Controller: manages cluster metadata, broker membership and leader elections. Since Kafka 4.0, this is always KRaft: a small quorum of controller nodes (3 or 5) replicating a metadata log with Raft. ZooKeeper support was removed in 4.0.
04
Clients bootstrap from any broker, fetch metadata (which broker leads each partition), and then talk directly to leaders. When leadership changes, clients refresh metadata.
05
Consumer group: a set of consumers sharing a group.id; each partition is assigned to exactly one member, so a group processes a topic in parallel while each partition's order is preserved. A group coordinator (a broker) manages membership and stores committed offsets in the internal __consumer_offsets topic.

Code & diagrams

cluster.mermaiddiagram

Rendering diagram…

describe-topic.shbash

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
Topic: orders  PartitionCount: 3  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Topic: orders  Partition: 0  Leader: 1  Replicas: 1,3,2  Isr: 1,3,2
  Topic: orders  Partition: 1  Leader: 2  Replicas: 2,1,3  Isr: 2,1,3
  Topic: orders  Partition: 2  Leader: 3  Replicas: 3,2,1  Isr: 3,2,1

kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
LeaderId:               1001
LeaderEpoch:            7
HighWatermark:          48211
CurrentVoters:          [1001,1002,1003]

Explain it without notes

Name every component a produced record touches from the producer to a consumer.

Practice

In the lab (next topic), create a topic with 3 partitions and describe it. Which broker leads each partition?

Trade-offs

↔
More brokers spread load and failure risk but add network replication and operational work.

Done when you can

I can explain brokers, topics, partitions, offsets, replicas, leaders, ISR, controllers and consumer groups.