Kafka Deep Dive

In one line

The distributed commit log: topic, partition, offset, consumer group, producer, consumer, replication — the full mental model.

0/16 · 0%

Think of it like this

A very durable, replayable news ticker tape that a company keeps forever. Once a headline is printed on the tape, any number of readers can walk up and read it (even hours later), and the tape itself never forgets what was printed.

Key ideas

01
The chain to internalize: Producer → Topic → Partition → Broker → Consumer Group → Offset (see the Blog for the full essay).
02
Append-only, ordered log per partition; replication factor N → each partition has N copies across brokers.
03
Why partitions: parallel throughput — N partitions allow N concurrent writers and N consumers per group.
04
Ordering: strictly per-partition; keys hash to partitions (same key = same partition = ordered).
05
Consumer groups split partitions — max active consumers = partition count.
06
Consumer failure → rebalance → some partitions move to survivors → reprocessing from old offsets → DUPLICATES (design consumers idempotent).
07
Producer acks: 0 (fire), 1 (leader), all (ISR quorum) — the durability dial.
08
Retention: Kafka keeps the log (configurable days); consumers replay from any offset — the property NoSQL queues don't have.

Java / Spring map

→
Spring Kafka: @KafkaListener(groupId=...), ProducerFactory config acks=all + retries; ConsumerSeekAware to replay.

Code & diagrams

topics, partitions, consumer groupsdiagram

Rendering diagram…

SpringKafka.javajava

Producer + consumer skeleton — the shapes you'll be drawing in interviews.

// --- producer ---
@Configuration
public class KafkaProducerConfig {
  @Bean
  public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> p = new HashMap<>();
    p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
    p.put(ProducerConfig.ACKS_CONFIG, "all");        // durability dial
    p.put(ProducerConfig.RETRIES_CONFIG, 5);         // at-least-once + dupes possible
    p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    return new DefaultKafkaProducerFactory<>(p);
  }
  @Bean public KafkaTemplate<String, String> kafkaTemplate() { return new KafkaTemplate<>(producerFactory()); }
}

// --- consumer (idempotent by design) ---
@Service
public class OrderEventConsumer {
  @KafkaListener(topics = "order-events", groupId = "payment-service")
  public void on(String message) {
    var event = OrderEvent.parse(message);
    if (paymentRepo.existsByEventId(event.eventId)) return;   // idempotency
    paymentRepo.process(event);                                // upsert by event id
  }
}

Explain without notes

Consumer dies mid-batch → explain rebalance, offset reset, and the duplicate that follows.

Broker dies → ISR, leader election, and the durability the producer asked for with acks=all.

Practice

Sketch the full chain with 3 partitions, 4 consumers (one idle) and offsets.

Design the retry/DLQ topic layout for an event that always fails validation.

Trade-offs

↔
Exactly-once is possible (transactions + idempotent producer) but the standard production answer is at-least-once + idempotent consumers.

Run it in production

You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:

Stateful · Kafka fundamentals

Run Kafka, create partitioned topics, and see per-key ordering on disk.

Stateful · Consumer groups & delivery semantics

Rebalancing, offsets, at-least-once, idempotency, DLQs, and consumer lag.

Stateful · Kafka replication, retention & operations

ISR, min.insync.replicas, compaction, schemas, and Strimzi.

Completion checklist

I can reproduce the full Kafka mental model and the four failure Q&A from the PDF (consumer dies, broker dies, ordering, duplicates).

Back to phase

Command Palette