Unit 2.1 · Kafka and Messaging

Kafka Fundamentals: Topics, Partitions, Offsets, Brokers

The log as a data structure, how topics split into partitions, what an offset is, how brokers and KRaft controllers fit together, and your first producer and consumer.

Beginner 40 min 4 lab steps 2 failure drills

Start here

The mental model

Kafka is not a queue where messages disappear once read. It's a set of append-only LOGS, like a ship's logbook: new entries are only added at the end, each has a line number (the OFFSET), and any number of readers can read the same book at their own pace, each keeping a bookmark. Entries are deleted by age or size (retention), not by being read.

A TOPIC is a named logbook (orders). Because one book can only be written so fast, a topic is split into PARTITIONS, several books written in parallel, spread across servers (BROKERS). Order is guaranteed within a partition, not across the whole topic.

Go deeper

How it works inside

01Topics, partitions, offsets

Each partition is an ordered, immutable sequence of records, stored on disk as SEGMENT files (by default 1 GB each, plus indexes). A record has a KEY (optional), a VALUE (bytes, often JSON, Avro, or Protobuf), headers, and a timestamp. The producer's partitioner hashes the key to choose the partition, so all events for order-1234 land in the same partition, in order. With no key, records are spread across partitions in batches.

An OFFSET is the position of a record in its partition (0, 1, 2…). Consumers track 'I've processed up to offset N in partition 3', and Kafka stores those committed offsets in an internal topic, __consumer_offsets. Rewinding is just moving the bookmark, which is how you replay events after fixing a bug.

Topics, partitions, offsetsdiagram

Rendering diagram…

02Brokers, controllers, and KRaft

A Kafka CLUSTER is several BROKERS; each partition has one LEADER broker that handles all reads and writes and FOLLOWER brokers that copy it (Unit 2.3). CONTROLLERS manage cluster metadata: which brokers are alive, who leads each partition, topic configs. Since Kafka 4.0 this runs on KRAFT, Kafka's own Raft quorum (usually 3 controller nodes). ZooKeeper, which older guides require, has been removed.

Clients bootstrap from any broker, download metadata (who leads which partition), then connect directly to the leaders. So every broker's ADVERTISED address must be reachable by clients, which is the most common Docker and Kubernetes networking pitfall with Kafka (Networking course, container networking).

03Why Kafka is fast

Sequential appends to disk (far faster than random writes), the OS page cache instead of an application cache, batching and compression of many records per request, and zero-copy transfer from page cache to socket (sendfile). A modest cluster handles hundreds of MB/s. The price is that Kafka is a low-level log: routing, retries, dead letters, and schemas are the application's job.

Do it

Hands-on lab

1
Run a single-node Kafka in KRaft mode
The official image starts a combined broker+controller with sensible defaults. The CLI tools live in /opt/kafka/bin.
terminal
$ docker run -d --name kafka -p 9092:9092 apache/kafka:4.1.0
docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
── expected output ──
Created topic orders.
Topic: orders TopicId: q2V... PartitionCount: 3 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: orders Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: orders Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: orders Partition: 2 Leader: 1 Replicas: 1 Isr: 1
2
Produce keyed events
parse.key=true with a : separator turns order-1:placed into key order-1, value placed.
terminal
$ printf 'order-1:placed\norder-2:placed\norder-1:paid\norder-3:placed\norder-1:shipped\n' | \
docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
--property parse.key=true --property key.separator=:
3
Consume and see per-key ordering
Print partition and offset with each record. All order-1 events are in one partition, in order; across partitions, output is interleaved.
terminal
$ docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning \
--property print.key=true --property print.partition=true --property print.offset=true --timeout-ms 5000
── expected output ──
Partition:0 Offset:0 order-2 placed
Partition:2 Offset:0 order-1 placed
Partition:2 Offset:1 order-1 paid
Partition:2 Offset:2 order-1 shipped
Partition:1 Offset:0 order-3 placed
Processed a total of 5 messages
4
Look at the log on disk
Each partition is a directory of segment files. This is literally the log.
terminal
$ docker exec kafka ls /tmp/kafka-logs/orders-2/
── expected output ──
00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex
leader-epoch-checkpoint
partition.metadata

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
num.partitions	1	Default partitions for auto-created topics.	Disable auto-creation (`auto.create.topics.enable=false`) and set partitions explicitly per topic.
retention.ms	604800000 (7 days)	How long records are kept.	Match replay needs and disk; use compaction for 'latest value per key' topics (Unit 2.3).
segment.bytes	1 GiB	Size at which a new segment file starts.	Smaller segments let retention/compaction act sooner on low-volume topics.
advertised.listeners	listeners value	Addresses the broker tells clients to connect to.	Must be reachable from every client network (separate listeners for in-cluster and external).

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Connects, then times out

A developer runs Kafka in Docker Compose and a consumer on the laptop. The bootstrap connection works, then everything hangs.

terminal

$ python consumer.py

── what you'll see ──

WARN [Consumer clientId=..] Connection to node 1 (kafka/172.19.0.4:9092) could not be established. Node may not be available.

Drill #2

Order events processed out of order

The payments team sees paid processed before placed for some orders.

terminal

$ kafka-console-consumer.sh ... --property print.key=true --property print.partition=true | grep order-981

── what you'll see ──

Partition:4 null {"order":"order-981","event":"paid"}

Partition:1 null {"order":"order-981","event":"placed"}

The bigger picture

Connects to

System Design · Kafka Deep Dive

The distributed commit log: topic, partition, offset, consumer group, producer, consumer, replication — the full mental model.

System Design · Partitioning

Splitting one logical dataset into smaller slices so each slice can live (and be queried) on its own node.

Networking · Container networking

Why advertised listeners break across Docker and Kubernetes networks.

AWS · SNS & EventBridge

The managed event-bus alternatives (Unit 2.4 compares them).

Prove it

Interview questions

How is Kafka different from a traditional message queue?

How does Kafka guarantee ordering?

What is KRaft?

0/4 · 0%

Command Palette