Command Palette

Search for a command to run...

Hectal
Phase 2Intermediate3 of 18 in Apache Kafka

Producers

Inside the producer: serializers, partitioner, record accumulator and sender thread; batching and compression; acks and durability; retries, timeouts, the idempotent producer and in-flight requests.

Most data loss and duplication in Kafka systems is decided at the producer: which acks setting, what happens on a timeout, whether retries can duplicate or reorder records. Most throughput problems are also decided here, by batching and compression.

Since Kafka 3.0 the defaults are safe (acks=all, idempotence on). This phase explains what those defaults do, when they're not enough, and how to tune for throughput without giving up safety.

0/4 · 0%
4 topics ~33 min 8 code blocks & diagrams
Start with the first topic
1
2.1

Producer Architecture: From send() to the Broker

send() is asynchronous: the record is serialized, assigned a partition, appended to a per-partition batch in the RecordAccumulator, and later shipped by a background sender thread in produce requests grouped per broker. Acknowledgements come back through the returned future or callback.

8 min 1 diagram 1 code practice

2
2.2

Batching, linger.ms, and Compression

Producers send records in per-partition batches; batch.size caps a batch's bytes and linger.ms waits a little for it to fill. Bigger batches compress better and cut requests, raising throughput at a small latency cost. Compression (lz4, zstd, snappy, gzip) trades CPU for network and disk.

7 min 2 code practice

3
2.3

acks and Producer Durability

acks decides when a write counts as successful: 0 (don't wait), 1 (leader wrote it), all (all in-sync replicas have it). Only acks=all combined with replication factor ≥ 3 and min.insync.replicas ≥ 2 makes an acknowledged record survive a broker failure.

9 min 1 diagram 1 code practice

4
2.4

Retries, Timeouts, the Idempotent Producer, and In-Flight Requests

When a produce request times out, the producer can't know whether the broker wrote the batch, so retrying can duplicate it. The idempotent producer (on by default since Kafka 3.0) attaches a producer ID and per-partition sequence numbers so brokers drop duplicates and keep order even with up to 5 in-flight requests.

9 min 1 diagram 1 code practice