Producers, Consumer Groups, Rebalancing, and Delivery Semantics

1. 1

   Create a 6-partition topic and start two consumers in one group

   Run each consumer in its own terminal. Then describe the group to see who owns what.

   terminalcopy cmd

   $ docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic payments --partitions 6

   # terminal A and B:

   docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments --group email

   # terminal C:

   docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group email

   ── expected output ──

   GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID

   email payments 0 0 0 0 console-consumer-5c1e...

   email payments 1 0 0 0 console-consumer-5c1e...

   email payments 2 0 0 0 console-consumer-5c1e...

   email payments 3 0 0 0 console-consumer-a93b...

   email payments 4 0 0 0 console-consumer-a93b...

   email payments 5 0 0 0 console-consumer-a93b...

2. 2

   Create lag and watch it drain

   Stop both consumers, produce 100,000 records with the perf tool, and describe the group: the lag is exactly the unread records. Restart a consumer and watch LAG fall to 0.

   terminalcopy cmd

   $ docker exec kafka /opt/kafka/bin/kafka-producer-perf-test.sh --topic payments --num-records 100000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=all

   docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group email | awk '{s+=$6} END {print "total lag:", s}'

   ── expected output ──

   100000 records sent, 61349.7 records/sec (11.70 MB/sec), 212.33 ms avg latency, 498.00 ms max latency.

   total lag: 100000

3. 3

   Replay by resetting offsets

   A bug in the email consumer sent wrong emails for the last hour. Fix it, stop the group, and rewind. --dry-run shows the plan first; --execute applies it. Resetting only works while the group has no active members.

   terminalcopy cmd

   $ docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group email --topic payments \

   --reset-offsets --to-datetime 2026-09-27T09:00:00.000 --execute

   ── expected output ──

   GROUP TOPIC PARTITION NEW-OFFSET

   email payments 0 14520

   email payments 1 14493

   ...

4. 4

   A safe consumer loop (Python)

   At-least-once with manual commits, a DLQ for poison messages, and an idempotency check. Here the language matters a little: the same structure applies in Java (Spring Kafka's DefaultErrorHandler + DeadLetterPublishingRecoverer does it for you).

   from confluent_kafka import Consumer, Producer
   import json
   
   c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "email",
                 "enable.auto.commit": False, "auto.offset.reset": "earliest",
                 "partition.assignment.strategy": "cooperative-sticky"})
   dlq = Producer({"bootstrap.servers": "localhost:9092"})
   c.subscribe(["payments"])
   
   while True:
       msg = c.poll(1.0)
       if msg is None or msg.error():
           continue
       try:
           event = json.loads(msg.value())
           if not already_sent(event["payment_id"]):      # idempotency key
               send_receipt(event)
               mark_sent(event["payment_id"])
       except Exception as e:                              # poison message
           dlq.produce("payments.dlq", msg.value(), key=msg.key(),
                       headers={"error": str(e), "src.offset": str(msg.offset())})
           dlq.flush()
       c.commit(message=msg, asynchronous=False)          # after processing

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.