Your Kafka Lab: KRaft in Docker, CLI Tools, and kcat

In one line

Run Kafka 4.x in KRaft mode with Docker (single node, then a 3-broker cluster), and learn the tools you'll use all course: kafka-topics, the console producer and consumer, kafka-consumer-groups, kafka-configs and kcat.

0/5 · 0%

Think of it like this

A practice kitchen. You learn knife skills where burning a dish costs nothing. Your lab is where you'll kill brokers, reset offsets and fill disks on purpose.

Key ideas

01
Single node: docker run -d -p 9092:9092 apache/kafka:4.1.0 starts a combined broker+controller in KRaft mode. Tools live in /opt/kafka/bin inside the container.
02
Three brokers: a Compose file with three combined nodes (or three controllers plus three brokers) lets you test replication, leader election and min.insync.replicas. Set KAFKA_NODE_ID, KAFKA_PROCESS_ROLES, KAFKA_CONTROLLER_QUORUM_VOTERS and advertised listeners per node.
03
Everyday commands: create and describe topics; produce with keys (--property parse.key=true --property key.separator=:); consume from the beginning with keys, partitions and offsets printed; describe consumer groups to see lag; reset offsets (--reset-offsets --to-datetime ... --execute); alter topic configs with kafka-configs.
04
kcat (formerly kafkacat) is a fast, scriptable client: kcat -b localhost:9092 -t orders -C -f '%p %o %k %s\n' prints partition, offset, key and value. Great for debugging.
05
Pin versions to match production (clients and brokers). Kafka 4.0 requires Java 17 for brokers and Java 11+ for clients, and removed ZooKeeper mode and some old client protocols, so very old clients may not connect.

Code & diagrams

docker-compose.ymlyaml

Three combined KRaft nodes. Good enough to practise replication and failover.

x-kafka: &kafka
  image: apache/kafka:4.1.0
  environment: &env
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
    CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

services:
  kafka1:
    <<: *kafka
    ports: ["19092:19092"]
    environment:
      <<: *env
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:19092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,EXTERNAL://localhost:19092
  kafka2:
    <<: *kafka
    ports: ["29092:29092"]
    environment:
      <<: *env
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092,EXTERNAL://localhost:29092
  kafka3:
    <<: *kafka
    ports: ["39092:39092"]
    environment:
      <<: *env
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:39092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092,EXTERNAL://localhost:39092

cli.shbash

B=localhost:19092
kafka-topics.sh --bootstrap-server $B --create --topic orders --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2
Created topic orders.

kafka-console-producer.sh --bootstrap-server $B --topic orders \
  --property parse.key=true --property key.separator=:
> order-9:{"status":"CREATED"}
> order-9:{"status":"PAID"}

kafka-console-consumer.sh --bootstrap-server $B --topic orders --from-beginning --group demo \
  --property print.key=true --property print.partition=true --property print.offset=true
Partition:4	Offset:0	order-9	{"status":"CREATED"}
Partition:4	Offset:1	order-9	{"status":"PAID"}

kafka-consumer-groups.sh --bootstrap-server $B --describe --group demo
GROUP TOPIC  PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
demo  orders 4         2              2              0   console-consumer-...

kcat -b $B -t orders -C -f 'p=%p o=%o k=%k %s\n' -e

When it breaks

Advertised listeners point to an address clients can't reach

What you see

Clients connect to the bootstrap server, receive metadata listing brokers as kafka1:9092, then fail to connect from outside Docker: timeouts and UNKNOWN_TOPIC style errors that look unrelated.

Fix & prevent

Advertise an address reachable by each client network (separate internal and external listeners).

Explain it without notes

Why does Kafka need separate listeners and advertised listeners?

Practice

Start the 3-node lab, create orders with RF 3, produce keyed records, stop kafka1, and describe the topic again.

Trade-offs

↔
Combined broker+controller nodes are simple for labs and small clusters; production clusters usually run dedicated controllers.

Done when you can

I have a working single-node and 3-node KRaft lab.
I can create topics, produce keyed records, consume with metadata, inspect groups and reset offsets.

Command Palette