Replication, ISR, Retention, Compaction, and Running Kafka

1. 1

   A three-broker cluster with Strimzi

   On a kind cluster, install the Strimzi operator, then declare the cluster. Node pools define broker and controller roles.

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: dual
     labels: { strimzi.io/cluster: shop }
   spec:
     replicas: 3
     roles: [controller, broker]
     storage: { type: persistent-claim, size: 5Gi, deleteClaim: false }
   ---
   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: shop
   spec:
     kafka:
       listeners:
         - { name: plain, port: 9092, type: internal, tls: false }
       config:
         default.replication.factor: 3
         min.insync.replicas: 2
         unclean.leader.election.enable: false
         auto.create.topics.enable: false
     entityOperator: { topicOperator: {}, userOperator: {} }
   ---
   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaTopic
   metadata:
     name: orders
     labels: { strimzi.io/cluster: shop }
   spec:
     partitions: 6
     replicas: 3
     config: { retention.ms: 604800000 }

   terminalcopy cmd

   $ kubectl create ns kafka && kubectl -n kafka apply -f https://strimzi.io/install/latest?namespace=kafka

   kubectl -n kafka apply -f kafka-cluster.yaml && kubectl -n kafka wait kafka/shop --for=condition=Ready --timeout=600s

   kubectl -n kafka exec shop-dual-0 -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders | head -3

   ── expected output ──

   kafka.kafka.strimzi.io/shop condition met

   Topic: orders PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2,retention.ms=604800000

   Topic: orders Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2

   Topic: orders Partition: 1 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0

2. 2

   Kill a broker, keep writing

   Delete one broker pod while producing. Leadership moves, the ISR shrinks to two, and acks=all writes keep succeeding. When the pod returns it catches up and rejoins the ISR.

   terminalcopy cmd

   $ kubectl -n kafka delete pod shop-dual-1

   kubectl -n kafka exec shop-dual-0 -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders --under-replicated-partitions

   ── expected output ──

   Topic: orders Partition: 1 Leader: 2 Replicas: 1,2,0 Isr: 2,0

   Topic: orders Partition: 4 Leader: 0 Replicas: 1,0,2 Isr: 0,2

3. 3

   A compacted topic as a table

   Product prices keyed by SKU. After compaction runs, only the last price per SKU remains. Low segment settings make compaction happen quickly in the lab.

   terminalcopy cmd

   $ kubectl -n kafka exec -i shop-dual-0 -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic prices \

   --config cleanup.policy=compact --config segment.ms=10000 --config min.cleanable.dirty.ratio=0.01

   printf 'sku-1:499\nsku-2:120\nsku-1:449\nsku-1:399\n' | kubectl -n kafka exec -i shop-dual-0 -- bin/kafka-console-producer.sh \

   --bootstrap-server localhost:9092 --topic prices --property parse.key=true --property key.separator=:

   # ~1 minute later, after a new segment rolls:

   kubectl -n kafka exec shop-dual-0 -- bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic prices --from-beginning --property print.key=true --timeout-ms 5000

   ── expected output ──

   sku-2 120

   sku-1 399

   Processed a total of 2 messages

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.